ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse

# ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
Source: [https://arxiv.org/html/2605.22850](https://arxiv.org/html/2605.22850)
###### Abstract\.

Prefix KV caching has become a key mechanism in LLM serving: it reduces time to first token \(TTFT\) by avoiding redundant computation across requests that share a prefix \(i\.e\., the system prompt\)\. However, the accumulated KV cache is often larger than what GPU memory and local DRAM can hold\. To preserve latency, current systems keep the KV cache in remote DRAM pools, increasing serving\-cluster size and cost\. In this paper, we explore a different approach: storing the KV cache in S3\-compatible object storage so that capacity is no longer the constraint, while minimizing the impact on TTFT\. We propose ObjectCache, which co\-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it, overlapping data transfer with compute across concurrent requests\. We prototype ObjectCache on a 100 Gbps RoCE cluster with NIXL \(an inference library that abstracts storage and memory\), Ceph RGW \(an Object Gateway for clusters\), and DAOS \(an open source storage system\)\. For 64K contexts, common in today’s systems, ObjectCache adds only 5\.6% latency over local DRAM; for 4K contexts, where less compute is available to mask transfer, ObjectCache adds 56–75 ms over the optimal local layerwise baseline\. Under shared bandwidth caps, our scheduler reduces added TTFT by 1\.2–1\.8x compared with equal bandwidth sharing\.

## 1\.Introduction

Modern LLM serving increasingly relies on long and reusable input contexts, including system prompts, retrieval\-augmented generation, code repositories, long conversations, and agent histories\(Xianget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib106); Wanget al\.,[2025b](https://arxiv.org/html/2605.22850#bib.bib107); Fanet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib108); Abou Aliet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib109)\)\(Figure[1](https://arxiv.org/html/2605.22850#S1.F1)\)\. These workloads make the prefill phase expensive because the model must materialize key–value \(KV\) tensors for every token and every layer before decoding can begin\. Prefix KV\-cache reuse avoids recomputing the shared prefix across requests and is therefore a primary mechanism for reducing TTFT\. As context lengths and reuse opportunities grow, however, the aggregate KV\-cache footprint that a serving cluster must retain also grows rapidly \(Appendix Figure[A1](https://arxiv.org/html/2605.22850#A2.F1)\)\.

The challenge is that reusable KV cache is much larger than the memory capacity naturally available near GPUs\. GPU memory is scarce and needed for model weights and active requests, while local DRAM only scales with the serving nodes that happen to produce or consume the cache\. Recent systems therefore use remote DRAM pools or memory\-centric KV\-cache tiers to preserve low latency\(Qinet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib1); Huet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib11)\)\. Although effective, this approach makes cache capacity a provisioned part of the serving cluster: memory must be sized for the retained KV working set and remains coupled to the compute deployment, thus increasing cost and limiting how much long\-lived reusable context the system can retain\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x1.png)Figure 1\.Long\-context LLM tasks increasingly reuse long\-lived prefixes, growing the aggregate KV cache footprint that a serving cluster must retain\.This motivates a different question: can S3\-compatible object storage serve as a runtime backend for reusable KV cache? Object storage is attractive because prefix KV blocks are immutable after prefill, naturally addressable by content\-derived prefix hashes, and reusable across users, sessions, and compute nodes\. Unlike a remote\-DRAM tier, an object store scales capacity independently of the GPU serving cluster and persists cache state beyond the lifetime of any particular worker\. If object storage could be used on the serving path, prefill and decode workers would become largely stateless with respect to reusable prefixes: once KV blocks are committed, later requests could fetch them from the shared object tier rather than returning to the node that produced them\(Zhonget al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib32); Patelet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib33)\)\.

The obstacle is not only transport bandwidth\. Even with an RDMA data path like Nvidia cuObject\(NVIDIA,[2026b](https://arxiv.org/html/2605.22850#bib.bib2),[a](https://arxiv.org/html/2605.22850#bib.bib3)\), the standard S3 abstraction remains mismatched to runtime KV\-cache reuse\. S3 exposes object\-level operations: a request names one object, or one byte range of one object\. In contrast, a prefix hit returns a set of many hash\-addressed KV chunks, and the inference engine consumes the matched cache in layer order\. Fetching each chunk or each layer range with independent S3 requests exposes fixed request overhead on the TTFT critical path, while fetching coarse objects sacrifices fine\-grained prefix reuse\. Thus, the gap between S3 and runtime KV\-cache reuse is semantic: object storage moves objects, whereas inference needs prefix\-selected KV blocks delivered in the order the GPU consumes them\.

We present ObjectCache, a protocol\-scheduling co\-design that makes S3\-compatible object storage suitable for runtime KV\-cache reuse\. On the protocol side, ObjectCache keeps KV cache stored as fine\-grained, hash\-addressed chunks, but extends the S3\-compatible request with a compact descriptor that names the matched chunks, model layout, delivery order, and RDMA target\. The storage server uses this descriptor to gather many chunk ranges, assemble one layer\-major payload at a time, and deliver each layer directly to the serving node over RDMA\. On the scheduling side, ObjectCache exploits the fact that layerwise transfer can overlap with per\-layer GPU computation\. Rather than sharing bandwidth equally or in proportion to bytes, ObjectCache allocates bandwidth according to each request’s per\-layer stall target, avoiding bandwidth assignments that do not further reduce TTFT\.

We prototype ObjectCache on a 100 Gbps RoCE cluster using NIXL\(Nvidia,[2026](https://arxiv.org/html/2605.22850#bib.bib47)\), Ceph RGW\(Ceph,[2026](https://arxiv.org/html/2605.22850#bib.bib43)\), and DAOS\(DAOS,[2026](https://arxiv.org/html/2605.22850#bib.bib42)\), and evaluate it with Llama 3\.1 8B\(Meta,[2026](https://arxiv.org/html/2605.22850#bib.bib38)\)\. For 64K\-token contexts, ObjectCache adds only 5\.6% TTFT over an optimized local layerwise baseline\. For shorter 4K\-token contexts, where less compute is available to hide transfer, ObjectCache adds 56–75 ms over the local layerwise baseline\. Under shared bandwidth caps, the ObjectCache scheduler reduces added TTFT by 1\.2–1\.8×\\timescompared with equal bandwidth sharing\.

This paper makes the following contributions:

- •We propose ObjectCache, a protocol\-scheduling co\-design for serving reusable KV cache from S3\-compatible object storage\.
- •We design an S3\-compatible protocol extension that supports prefix\-selected multi\-object aggregation and layerwise KV delivery while preserving fine\-grained client\-side prefix lookup\.
- •We introduce a bandwidth scheduler that allocates storage bandwidth according to per\-layer compute\-transfer overlap, reducing TTFT under contention\.
- •We demonstrate on a 100 Gbps RoCE prototype that ObjectCache approaches local layerwise KV\-cache performance for long\-context workloads and improves TTFT under shared bandwidth limits\.

## 2\.Background and Motivation

![Refer to caption](https://arxiv.org/html/2605.22850v1/x2.png)Figure 2\.Per\-layer KV payload for a 16\-token chunk across recent open\-weight LLM families\. The 64 KB dashed line marks the grouped\-query attention \(GQA\) baseline with 8 KV heads of 128 dimensions; multi\-head latent attention \(MLA\) and smaller head counts push recent models below this threshold\.### 2\.1\.KV cache and Prefix Reuse

During the prefill phase, LLM inference processes the input prompts and materializes the per\-layer key/value tensors to be used in transformer attention\(Vaswaniet al\.,[2017](https://arxiv.org/html/2605.22850#bib.bib35); Daoet al\.,[2022](https://arxiv.org/html/2605.22850#bib.bib36)\)\. During the decode phase, the model reuses these tensors instead of recomputing attention over the entire prefix each time by storing the context in a KV cache\. To enable cross\-query reuse and storage, serving systems partition the KV cache into fixed\-size chunks corresponding toGGconsecutive tokens\. For a model withLLlayers,nkvn\_\{\\mathrm\{kv\}\}KV heads, head dimensiondd, and element widthpp, the KV bytes per token and the per\-layer bytes of aGG\-token chunk are:

\(1\)KVtoken=2​L​nkv​d​p,Slayer,chunk=2​G​nkv​d​p\.\\mathrm\{KV\}\_\{\\mathrm\{token\}\}=2Ln\_\{\\mathrm\{kv\}\}dp,\\qquad S\_\{\\mathrm\{layer,chunk\}\}=2Gn\_\{\\mathrm\{kv\}\}dp\.The prefix KV cache reuse relies on a radix tree or similar prefix index to find the longest cached match for a new request\(Kwonet al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib13); Yeet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib17); Gimet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib16); Gaoet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib15); Wanget al\.,[2025a](https://arxiv.org/html/2605.22850#bib.bib7)\)\. Each chunk can be identified by a rolling hash,Hi=Hash​\(Hi−1∥tokensi\)H\_\{i\}=\\mathrm\{Hash\}\(H\_\{i\-1\}\\parallel\\mathrm\{tokens\}\_\{i\}\), that gives it a deterministic object key\. UsingHiH\_\{i\}as the object key gives KV chunks the properties object storage is good at: immutable writes, content\-addressed deduplication, and independent reuse across requests\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x3.png)\(a\)Fine granularity preserves intermediate branch points\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x4.png)\(b\)Coarse granularity merges branch points\.

Figure 3\.Prefix reuse under fine and coarse storage granularities\. Coarse chunks reduce index depth but lose branch points where requests can diverge, forcing otherwise reusable tokens to be recomputed\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x5.png)Figure 4\.Prefix\-hash lookup cost is small relative to tokenization, even at 16\-token granularity\. Fine\-grained indexing is therefore not the request critical\-path bottleneck\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x6.png)Figure 5\.ObjectCache in a disaggregated cluster\. Prefix KV caches are stored in an object\-storage tier through an S3\-compatible interface, decoupling prefill and decode workers from the machines that produced the cache\. ObjectCache extends this interface so object storage can serve KV cache reuse with the granularity and layerwise delivery order expected by LLM serving systems\.At fine chunk sizes, the per\-layer KV payload becomes small enough that the fixed object\-request overhead is visible even when the communication overhead is small \(Figure[2](https://arxiv.org/html/2605.22850#S2.F2)instantiates Equation[1](https://arxiv.org/html/2605.22850#S2.E1)for current models\)\. A 16\-token chunk exposes only tens of KB per layer for many current models, and MLA\-style layouts can reduce the layer slice to roughly 16 KB\(Liuet al\.,[2024a](https://arxiv.org/html/2605.22850#bib.bib24); Menget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib103)\)\. This pressure remains even at coarser chunk sizes\. For Llama 3\.1 8B\(Meta,[2026](https://arxiv.org/html/2605.22850#bib.bib38)\), a 256\-token chunk is about 1 MB per layer; compact KV layouts, MLA\-style representations, or shape\-preserving compression can push that below the efficient object\-transfer regime\(Changet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib104); Zandiehet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib80)\)\. Increasing the chunk granularity enables larger physical data transfers, but also coarsens prefix reuse\. ObjectCache instead keeps the logical reuse granularity independent of the effective transfer granularity by aggregating chunks at the storage server\.

Figures[3](https://arxiv.org/html/2605.22850#S2.F3)and[4](https://arxiv.org/html/2605.22850#S2.F4)separate two concerns that are often conflated\. Figure[3](https://arxiv.org/html/2605.22850#S2.F3)shows why fine chunks preserve more radix\-tree branch points and avoid unnecessary recompute after a request diverges\. Figure[4](https://arxiv.org/html/2605.22850#S2.F4)shows that prefix\-hash lookup remains small compared to tokenization, even atG=16G\{=\}16granularity\. Fine\-grained indexing does not add meaningful overhead to the request critical path\. The system bottleneck is therefore not finding the prefix; it is delivering the matched KV chunks in the order the model can consume\.

### 2\.2\.Serving\-Path KV Access Pattern

In disaggregated serving, a prefill worker materializes KV cache for the input prompt, while a decode worker later uses the resulting KV state to generate tokens\. When a new request shares a cached prefix, prefix lookup returns an ordered list of matched KV chunks that the prefill worker can reuse before computing the remaining suffix\. Although these chunks are stored independently, the model consumes them layer by layer: it needs the layer 0 slice from all matched chunks before computing layer 0, then the layer 1 slice before computing layer 1, and so on\. Thus, the storage\-facing access pattern is an ordered multi\-object, multi\-range read rather than a single object read\.

## 3\.ObjectCache Design

Figure[5](https://arxiv.org/html/2605.22850#S2.F5)shows how ObjectCache fits into a disaggregated LLM serving cluster\. A central orchestrator receives a request, performs prefix matching, and assigns the remaining prefill work to a prefill node together with the matched prefix KV list\. The prefill node fetches the reusable KV cache from the S3\-compatible object tier, computes the remaining prompt suffix, and materializes the full KV state\. The decode node can then load the full KV state and generate output tokens, while newly produced KV blocks are offloaded back to object storage for future reuse\. This organization decouples prefill and decode workers from the machines that originally produced the cache, making reusable prefix state persistent and shared through the object\-storage tier\.

The key challenge is that this serving path needs more than ordinary object reads\. ObjectCache is built around one design rule:*keep storage objects small enough for prefix reuse, but expose a serving interface that returns large, layer\-ordered transfers*\. The design is therefore an interface extension, not a new transport\. The underlying S3\-over\-RDMA data path moves bytes; ObjectCache changes what the serving system can ask the object tier to do\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x7.png)Figure 6\.ObjectCache system design\. HTTP preserves S3\-compatible control, while the storage server aggregates matched chunk ranges into layerwise KV payloads and delivers them to the serving node over RDMA\.The resulting architecture has three roles \(Figure[6](https://arxiv.org/html/2605.22850#S3.F6)\)\. The*GPU serving node*issues S3\-compatible requests and consumes KV cache data through the inference engine’s layerwise interface\. The*gateway*terminates the S3 control plane, parses the ObjectCache descriptor, forwards the multi\-object request to the storage server, and upgrades the S3 control plane to express prefix\-aware, layer\-scheduled KV transfer\. The*storage server*resolves chunk objects in the S3 namespace, performs range reads, assembles layer\-major payloads, RDMA\-writes them directly to the client buffer, and executes the layer\-scheduled semantic at the object/range level\. All runtime policy—whether a given request is served chunkwise or via layerwise aggregation, and how concurrent tenants share the network cap—runs on the storage server, so the gateway and NIXL client both remain stateless with respect to scheduling decisions \(Sections[3\.4](https://arxiv.org/html/2605.22850#S3.SS4)and[3\.6](https://arxiv.org/html/2605.22850#S3.SS6)\)\.

Table 1\.ObjectCache request descriptor\.### 3\.1\.Required Semantics

ObjectCache requires four access semantics that no existing S3 primitive provides in combined form; this is the gap between accelerating S3 through better transport protocols and making S3 suitable for prefix KV reuse\.

\(1\) Multi\-object batching\.A prefix hit returns a list of matched KV chunks, not one object\. If each chunk requires an independent S3 request, fixed request overhead dominates in the fine\-grained regime that prefix caching prefers\.

\(2\) Layerwise delivery\.Inference consumes KV cache data layer by layer\. Delivering complete chunks in chunk\-major order forces the NIXL client to wait for the full matched prefix before starting layer 0\. Delivery of layer 0 for all chunks first creates a compute window to transfer later layers\.

\(3\) Hash\-derived immutable keys\.KV chunks must remain addressable by hashes so that requests reuse the same object when they share a prefix\. This preserves the radix\-tree storage semantics used by existing prefix caches\.

\(4\) Bulk\-transfer compatibility\.Once the layer\-major payload has been assembled, data should be transferred using RDMA because it provides the high\-throughput transport that the higher\-level aggregation semantics can use\.

Standard S3 GET satisfies hash\-derived keys but not batching or layerwise delivery\. Range\-GET can fetch one layer range, but only by issuing a request per chunk per layer\. S3\-over\-RDMA satisfies the bulk\-transfer requirement, but it inherits the same single\-object request model\. ObjectCache adds the missing server\-side multi\-object aggregation and layerwise delivery semantics while reusing the RDMA\.

### 3\.2\.The ObjectCache Descriptor

ObjectCache adds an S3\-compatible descriptor to a normal request\. The descriptor names the matched chunk keys, the model layout, the delivery order, and the RDMA target buffer\.

Table[1](https://arxiv.org/html/2605.22850#S3.T1)lists the fields of the descriptor, which is intentionally arithmetic rather than manifest\-heavy\. Because every chunk in the same model deployment has the same per\-layer sizeSS, the byte range for layerℓ\\ellin a chunk is\[ℓ​S,\(ℓ\+1\)​S\)\[\\ell S,\\;\(\\ell\+1\)S\)\. Thus, the storage server can derive all fixed\-shape layer ranges from the descriptor without a per\-object manifest\. Variable\-size or compressed layouts can add a manifest later, but the common serving fast path keeps the descriptor compact\.

### 3\.3\.Server\-Side Layer Aggregation

The storage server executes the descriptor by assembling one payload per model layer\. For each layerℓ∈\[0,L\)\\ell\\in\[0,L\), it range\-fetches\[ℓ​S,\(ℓ\+1\)​S\)\[\\ell S,\(\\ell\+1\)S\)from every matched chunk in parallel, appends the returned slices in prefix order, RDMA\-writes the assembled payload to the client buffer, and notifies the serving node that the layer is ready \(pseudocode in Appendix Table[A3](https://arxiv.org/html/2605.22850#A6.T3)\)\. This notification is on the critical path of inference: it lets the GPU start layer computation without waiting for the whole prefix to arrive\(Qinet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib1); Liuet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib105)\)\.

The physical storage layout remains chunkwise even though the logical delivery layout is layerwise\. In KV\_L2TD format \(Key\-Value tensors stored inLayer\-major order, with the2matrices concatenated per layer, thenToken position, then hiddenDimension\), each immutable prefix\-chunk object stores all layers sequentially, and each layer is arranged by token position and hidden dimension\. Server\-side aggregation changes only the readout order: one layerwise payload covers all matched chunks for a single model layer\.

Keeping storage fine\-grained matters because coarsening sacrifices reuse\. Larger KV blocks reduce object retrieval overhead but discard reuse when two requests diverge inside a block; increasing the lookup granularity from 16 to 512 tokens can force otherwise reusable tokens to be recomputed across models and hit boundaries \(Appendix Table[A6](https://arxiv.org/html/2605.22850#A6.T6)quantifies this recompute cost\)\. The right design point is therefore not “make objects as large as possible”; the system should keep lookup granularity small and transfer granularity large enough for efficient I/O\.

### 3\.4\.Server\-Side Mode Selection

ObjectCache does not serve every cache hit using server\-side layer aggregation\. The KV cache is always stored in the chunkwise layout above, so the storage server chooses only the delivery mode\. Each incoming descriptor names the matched chunks, from which the server derives the total matched payloadW=N⋅L⋅SW=N\\cdot L\\cdot S, whereNNis the number of matched chunks,LLis the number of model layers, andSSis the per\-layer chunk size\. It then applies a single size\-threshold rule:

\(2\)mode​\(W\)=\{chunkwiseW<Θ,layerwise \+ aggregationW≥Θ\.\\mathrm\{mode\}\(W\)=\\begin\{cases\}\\text\{chunkwise\}&W<\\Theta,\\\\ \\text\{layerwise \+ aggregation\}&W\\geq\\Theta\.\\end\{cases\}The thresholdΘ\\Thetais a deployment knob: payloads belowΘ\\Thetause chunkwise delivery, and larger payloads use server\-side aggregation with layerwise delivery\. BelowΘ\\Theta, the layerwise schedule of Equation[3](https://arxiv.org/html/2605.22850#S3.E3)still minimizes TTFT in theory, but the absolute saving is negligible relative to the aggregation overhead\. AboveΘ\\Theta, the aggregation pipeline recovers its CPU and I/O\-depth cost as measurable TTFT reduction while also amortizing per\-object descriptor overhead across many small range reads\.

As an illustration, on our 100 Gbps prototype with Llama 3\.1 8B we useΘ≈512\\Theta\\approx 512MB, the payload size at which network transfer time at line rate becomes comparable to the prefill compute window\. Under this rule, the 4K configurations in Section[5](https://arxiv.org/html/2605.22850#S5)fall on the chunkwise side and the 16K/64K configurations fall on the aggregated\-layerwise side, matching where each mode wins in Figure[13](https://arxiv.org/html/2605.22850#S5.F13)\. The appropriate value ofΘ\\Thetadepends on the storage backend, SSD queue depth, and link rate; the dispatch rule itself does not\. A full sensitivity analysis ofΘ\\Thetais left to future work\.

Equation[2](https://arxiv.org/html/2605.22850#S3.E2)is also what decides which requests enter multi\-tenant scheduling: chunkwise requests run independently against the object storage and consume only their fair share of object\-store concurrency, while every layerwise request joins the shared bandwidth pool described next\. The scheduling problem below is therefore scoped to layerwise requests under a shared bandwidth cap\.

### 3\.5\.Layerwise Prefetch

![Refer to caption](https://arxiv.org/html/2605.22850v1/x8.png)\(a\)Chunkwise loading waits for the full cached prefix before compute can start\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x9.png)\(b\)Transfer\-bound layerwise prefetch\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x10.png)\(c\)Compute\-bound layerwise prefetch\.

Figure 7\.KV cache fetch scheduling\. Chunkwise delivery serializes cache loading before prefill, while layerwise delivery exposes per\-layer readiness so transfer can overlap compute when bandwidth is sufficient and otherwise appears as per\-layer stall\.Figure[7](https://arxiv.org/html/2605.22850#S3.F7)explains why the delivery order matters\. A chunkwise baseline waits for all matched chunks before compute can use any layer from the KV cache\. Layerwise prefetch instead transfers layer 0 first; once layer 0 arrives, the GPU begins computing on it while the remaining layers transfer in parallel\. LetXℓX\_\{\\ell\}be the transfer time for layerℓ\\ellandCℓC\_\{\\ell\}be the compute time exposed by the miss tokens at layerℓ\\ell\. With one\-layer prefetch, the TTFT model is

\(3\)TTTFT≈X0\+∑ℓ=0L−2max⁡\(Xℓ\+1,Cℓ\)\+CL−1,T\_\{\\mathrm\{TTFT\}\}\\approx X\_\{0\}\+\\sum\_\{\\ell=0\}^\{L\-2\}\\max\(X\_\{\\ell\+1\},\\,C\_\{\\ell\}\)\+C\_\{L\-1\},whereX0X\_\{0\}is the latency before the GPU can start \(layer 0 must arrive in full\), the summation capturesL−1L\{\-\}1stages in which transfer and compute overlap, andCL−1C\_\{L\-1\}is the final layer’s compute after all transfers have finished\. WhenXℓ\>CℓX\_\{\\ell\}\>C\_\{\\ell\}, the transfer time that exceeds the compute window appears as additional waiting time in the corresponding stage of Equation[3](https://arxiv.org/html/2605.22850#S3.E3)\. Section[5\.3](https://arxiv.org/html/2605.22850#S5.SS3)evaluates this model with measured compute times and ObjectCache throughput\.

### 3\.6\.Bandwidth\-Aware Scheduling

Under the layerwise delivery of Section[3\.5](https://arxiv.org/html/2605.22850#S3.SS5), each request’s per\-layer transfer can overlap with its per\-layer compute window\. This creates a bandwidth\-scheduling problem specific to KV cache reuse: under a shared bandwidth capBB, policies proportional to matched bytes or equal\-share misallocate\. Matched\-bytes allocation over\-serves long\-prefix requests whose per\-layer transfer is already shorter than per\-layer compute, so extra bandwidth yields no further benefit, while equal\-share under\-serves short\-prefix requests whose compute window is small\.

The key observation is that each layer of a request transfers roughly the same KV bytes and performs similar compute, so we characterise requestiiby its per\-layer transfer sizesis\_\{i\}and per\-layer compute windowcic\_\{i\}\.111Both are approximately constant across layers because every layer has the same number of KV heads and the same attention/feed\-forward network \(FFN\) structure\.When the scheduler assigns bandwidthrir\_\{i\}to requestii, the per\-layer transfer time issi/ris\_\{i\}/r\_\{i\}\. Transfer that exceeds the compute window introduces additional stall:

\(4\)τi​\(ri\)=max⁡\(0,siri−ci\)\.\\tau\_\{i\}\(r\_\{i\}\)\\;=\\;\\max\\\!\\Bigl\(0,\\;\\frac\{s\_\{i\}\}\{r\_\{i\}\}\-c\_\{i\}\\Bigr\)\.
Table 2\.Interface\-level positioning\. Existing systems accelerate either the memory tier or the S3 byte path; ObjectCache adds KV\-aware multi\-object, layer\-major aggregation inside an S3\-compatible request\.Each request’s additional latency vanishes oncerir\_\{i\}reaches its*zero\-stall rate*ri∗=si/cir\_\{i\}^\{\*\}=s\_\{i\}/c\_\{i\}; any bandwidth beyond this point yields no further latency benefit\. In an unconstrained setting, every request would receiveri∗r\_\{i\}^\{\*\}, but the total demand∑iri∗\\sum\_\{i\}r\_\{i\}^\{\*\}may exceed the shared budgetBB\. The scheduler must then distribute a total cut of∑iri∗−B\\sum\_\{i\}r\_\{i\}^\{\*\}\-Bacross requests\. Letℛ\\mathcal\{R\}denote the set of active layerwise requests sharing the storage link, and letΔi≥0\\Delta\_\{i\}\\geq 0denote the bandwidth cut applied to requestii\(so its allocated rate isri=ri∗−Δir\_\{i\}=r\_\{i\}^\{\*\}\-\\Delta\_\{i\}\)\. Allocating these cuts to minimize total stall leads to the*Stall\-opt*problem:

\(5\)min\{Δi\}​∑i∈ℛ\(siri∗−Δi−ci\)s\.t\.∑i∈ℛΔi=∑i∈ℛri∗−B,0≤Δi≤ri∗\.\\min\_\{\\\{\\Delta\_\{i\}\\\}\}\\;\\sum\_\{i\\in\\mathcal\{R\}\}\\left\(\\frac\{s\_\{i\}\}\{r\_\{i\}^\{\*\}\-\\Delta\_\{i\}\}\-c\_\{i\}\\right\)\\quad\\text\{s\.t\.\}\\quad\\sum\_\{i\\in\\mathcal\{R\}\}\\Delta\_\{i\}=\\sum\_\{i\\in\\mathcal\{R\}\}r\_\{i\}^\{\*\}\-B,\\;0\\leq\\Delta\_\{i\}\\leq r\_\{i\}^\{\*\}\.
Sincecic\_\{i\}does not depend onΔi\\Delta\_\{i\}, substitutingri=ri∗−Δir\_\{i\}=r\_\{i\}^\{\*\}\-\\Delta\_\{i\}reduces the objective to minimising total transfer time under the bandwidth budget:

\(6\)min\{ri\}​∑i∈ℛsiris\.t\.∑i∈ℛri=B,0<ri≤ri∗\.\\min\_\{\\\{r\_\{i\}\\\}\}\\;\\sum\_\{i\\in\\mathcal\{R\}\}\\frac\{s\_\{i\}\}\{r\_\{i\}\}\\quad\\text\{s\.t\.\}\\quad\\sum\_\{i\\in\\mathcal\{R\}\}r\_\{i\}=B,\\;0<r\_\{i\}\\leq r\_\{i\}^\{\*\}\.
Becausesi/ris\_\{i\}/r\_\{i\}is convex inrir\_\{i\}, Equation[6](https://arxiv.org/html/2605.22850#S3.E6)is a convex program\. When∑iri∗≤B\\sum\_\{i\}r\_\{i\}^\{\*\}\\leq B, every request receives its zero\-stall rate and all additional TTFT vanishes; when the budget is tight, the optimal allocation admits a closed\-form solution\. In practice, the analyticalri∗r\_\{i\}^\{\*\}falls on the slope of the TTFT curve rather than at the start of the flat region \(see Section[5\.7](https://arxiv.org/html/2605.22850#S5.SS7)\)\.*Calibrated Stall\-opt*shifts the target by a small positive offsetδ\\deltato ensure the operating point lies on the plateau:

\(7\)r^i=ri∗\+δ,\\hat\{r\}\_\{i\}\\;=\\;r\_\{i\}^\{\*\}\+\\delta,
Calibrated Stall\-opt solves the bandwidth allocation of Equation[5](https://arxiv.org/html/2605.22850#S3.E5)with corrected targetsr^i\\hat\{r\}\_\{i\}as upper bounds\. When the total demand∑ir^i\\sum\_\{i\}\\hat\{r\}\_\{i\}exceeds the budgetBB, the closed\-form solution distributes the deficit to minimize total stall across concurrent requests \(pseudocode in Appendix Table[A4](https://arxiv.org/html/2605.22850#A6.T4)\)\. During each scheduling epoch, a batch of active layerwise requests is admitted under a fixed total bandwidth budget, and each request receives a stable target layer\-delivery rate for the duration of its KV load\. If one request finishes early, its released bandwidth returns to the pool for the next scheduling epoch rather than being redistributed immediately to in\-flight requests\. This conservative rule makes per\-request transfer times predictable, so the serving node does not need to react to unexpected bandwidth changes during an epoch\.

## 4\.Implementation

Our prototype integrates ObjectCache into a three\-node serving/storage stack\. The serving node runs the LLM and issues KV cache reads\. The gateway terminates S3\-compatible requests, parses ObjectCache descriptors, and forwards them to the storage server, which executes the multi\-object aggregation\. Concretely, the prototype uses NIXL as the asynchronous transfer library\(Nvidia,[2026](https://arxiv.org/html/2605.22850#bib.bib47)\), Ceph RGW as the S3\-compatible gateway\(Ceph,[2026](https://arxiv.org/html/2605.22850#bib.bib43)\), and DAOS as the storage server behind the gateway\(DAOS,[2026](https://arxiv.org/html/2605.22850#bib.bib42)\)\. The implementation keeps the gateway thin: the serving node issues normal S3\-compatible requests with additional headers, the gateway handles only S3 control, and the DAOS storage server performs the cross\-object range reads and layer\-major assembly\.

### 4\.1\.S3\-Compatible Paths

The prototype includes five S3\-compatible paths\. The first three isolate transport choices for a single object; the last two exercise the ObjectCache interface:

- •S3TCP: standard S3 request; object sent via HTTP/TCP\.
- •S3RDMA Buffer: S3 request naming one object; the gateway stages the payload before the RDMA transfer to the NIXL client\.
- •S3RDMA Direct: S3 request naming one object; the data goes through the object\-store RDMA path without gateway staging\.
- •S3RDMA Batch: S3 request naming multiple objects; one S3 request carries one HTTP header followed by an RDMA burst, thereby amortizing per\-object overhead\.
- •S3RDMA Agg: S3 request naming multiple objects; the storage server assembles layer\-major payloads and RDMA\-writes them to the NIXL client\.

We use the following names in the end\-to\-end TTFT evaluation:

- •Local\-DRAM\-CWandLocal\-DRAM\-LW: local CPU\-DRAM KV cache baselines with chunkwise and layerwise delivery, respectively\.
- •S3Batch\-CW: the S3\-backed chunkwise batched path\.
- •S3Agg\-LW: ObjectCache’s server\-side aggregated layerwise path\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x11.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x12.png)\(a\)READ,C=8C\{=\}8
![Refer to caption](https://arxiv.org/html/2605.22850v1/x13.png)\(b\)READ,C=32C\{=\}32
![Refer to caption](https://arxiv.org/html/2605.22850v1/x14.png)\(c\)WRITE,C=8C\{=\}8
![Refer to caption](https://arxiv.org/html/2605.22850v1/x15.png)\(d\)WRITE,C=32C\{=\}32

Figure 8\.Raw object\-storage interface baseline\. The gray region marks throughput above the 100 Gbps link capacity; points in this region are limited by local storage and host execution rather than the network\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x16.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x17.png)\(a\)S3 GET,C=8C\{=\}8
![Refer to caption](https://arxiv.org/html/2605.22850v1/x18.png)\(b\)S3 GET,C=32C\{=\}32
![Refer to caption](https://arxiv.org/html/2605.22850v1/x19.png)\(c\)S3 PUT,C=8C\{=\}8
![Refer to caption](https://arxiv.org/html/2605.22850v1/x20.png)\(d\)S3 PUT,C=32C\{=\}32

Figure 9\.S3\-compatible interface baseline\. S3RDMA Direct preserves high large\-object throughput, while S3TCP and S3RDMA Buffer expose protocol and staging bottlenecks\.
### 4\.2\.GPU Serving Node

The serving node uses the NIXL object backend to issue S3 operations\. For ordinary objects, the backend can use TCP, an RDMA\-buffer mode, or the RDMA\-direct path used in our microbenchmarks\. For ObjectCache requests, LMCache\(LMCache,[2026](https://arxiv.org/html/2605.22850#bib.bib44)\)provides the matched prefix chunk keys and the model layout parameters \(LL,GG, andSS\)\. NIXL serializes these fields into the HTTP descriptor and attaches an RDMA target token for the receive buffer\. The LLM inference framework, such as vLLM\(vLLM,[2026](https://arxiv.org/html/2605.22850#bib.bib45)\), waits for layer\-ready notifications and lets the model proceed as soon as the next layer’s KV has arrived\. This is the same execution pattern used by DRAM\-backed layerwise KV loading, but the source is now object storage\.

### 4\.3\.Gateway and Storage Server Execution

Ceph RGW is used as the gateway\. It parses the ObjectCache descriptor after normal S3 request handling, then forwards the descriptor to the DAOS storage server via a server\-side extension\. The DAOS server maps each chunk key to the corresponding DAOS object, fetches the per\-layer record extent from each chunk in parallel, concatenates the slices in chunk order, and RDMA\-writes the assembled layer payload directly into the client’s registered buffer\.

The gateway keeps the S3 abstraction in the control path: request headers, authentication, bucket/object naming, and access control are still handled at the S3 layer\. The data path is split: HTTP carries control, while RDMA carries the assembled layer payload directly between the storage server and the GPU serving node\. This mirrors the S3\-over\-RDMA control/data\-plane separation\(NVIDIA,[2026b](https://arxiv.org/html/2605.22850#bib.bib2),[a](https://arxiv.org/html/2605.22850#bib.bib3),[2025](https://arxiv.org/html/2605.22850#bib.bib49)\), but ObjectCache adds multi\-object range selection and layer\-major assembly at the storage server before the RDMA transfer\.

### 4\.4\.Tracing and Rate Control

To understand request overhead, the prototype propagates request identifiers from the NIXL client through RGW into DAOS traces, allowing the evaluation to split latency into client, gateway, storage, and RDMA\-bulk components\. For the multi\-tenant scheduling experiments, we add a layerwise pacing control that limits the effective layer delivery rate\. This lets us emulate shared bandwidth caps while keeping the same layerwise access pattern used by ObjectCache\.

### 4\.5\.Testbed

The evaluation uses a 100 Gbps RoCEv2 \(RDMA over Converged Ethernet, v2\) cluster with a GPU serving node for inference, an S3 gateway, and a DAOS storage node\. The GPU is an A100 80 GB GPU \(CUDA 12\.8, PyTorch 2\.10\.0, vLLM v0\.19\.0, the modified LMCache v0\.4\.2, the modified NIXL v1\.0\.0\); the gateway runs the modified Ceph RGW v18\.2\.7; the storage node runs DAOS v2\.7\.102 with its default UCX\(UCX,[2026](https://arxiv.org/html/2605.22850#bib.bib46)\)transport, equipped with 4 KIOXIA enterprise NVMe SSDs\. In this stack, vLLM is the inference runtime, LMCache manages KV cache reuse on the GPU serving node, NIXL provides the unified storage/memory transfer interface used by both the serving node and the gateway, Ceph RGW terminates S3 requests, DAOS is the underlying object\-storage backend, and UCX is the RDMA transport beneath DAOS\. DAOS stripes data across all 4 SSDs; however, because layerwise range reads access non\-contiguous offsets within each chunk object, the per\-SSD access pattern is random rather than sequential\. The measurement of the same micro\-benchmark for H100 80GB is in Appendix\.

## 5\.Evaluation

Our evaluation follows the same mechanisms introduced in the design\. First, the storage and network substrate must transfer data fast enough\. Second, the S3 control path must be measured separately from the RDMA data path\. Third, ObjectCache’s server\-side aggregation must achieve high throughput for fine\-grained KV chunks\. Fourth, layerwise delivery must overlap with GPU compute in configurations where measured compute and transfer rates make overlap feasible\. Finally, under a shared bandwidth cap, bandwidth\-aware scheduling should minimize TTFT\.

### 5\.1\.Raw Storage Baseline

#### Setup\.

We measure DAOS throughput as seen by the NIXL object client without the Ceph RGW gateway, isolating the storage backend from S3 protocol overhead \(Figure[8](https://arxiv.org/html/2605.22850#S4.F8)\)\. The benchmark ranges block sizes from 64 KB to 4 MB and client concurrencyC∈\{8,32\}C\\in\\\{8,32\\\}, whereC=8C\{=\}8uses a single thread with 8 in\-flight requests andC=32C\{=\}32uses 4 threads each with 8 in\-flight requests\. Writes populate deterministic per\-thread key spaces; a large scrub workload flushes the DAOS cache before measuring the cold\-read throughput\.

Local DAOS reads exceed the 100 Gbps NIC capacity at 256 KB block size withC=32C\{=\}32, and saturate the SSD throughput at 4 MB block size withC=32C\{=\}32\. DAOS over RDMA approaches the 100 Gbps hardware limit on cold reads at 1 MB block size withC=8C\{=\}8, while TCP lags consistently \(Figure[8](https://arxiv.org/html/2605.22850#S4.F8)\)\. We use the saturated RDMA configurations as the reference point for later S3 and ObjectCache experiments: any remaining gap is attributable to the S3 interface, gateway processing, GPU landing, or aggregation policy rather than the DAOS backend alone\.

### 5\.2\.S3 Transport Baseline

#### Setup\.

We next measure S3 PUT and GET throughput across block sizes from 64 KB to 4 MB\. The benchmark compares S3TCP, S3RDMA Buffer that uses a gateway staging buffer, and S3RDMA Direct where the S3 request carries control information data going through the DAOS RDMA data path; Section[4](https://arxiv.org/html/2605.22850#S4)gives the corresponding path names\. The two concurrency points,C=8C\{=\}8andC=32C\{=\}32, match the canonical raw\-storage configurations in Figure[8](https://arxiv.org/html/2605.22850#S4.F8)\. Figure[9](https://arxiv.org/html/2605.22850#S4.F9)shows that S3RDMA Direct approaches the NIC capacity at 4 MB block size withC=32C\{=\}32\. S3TCP and S3RDMA Buffer suffer from different bottlenecks\. S3TCP is limited by the gateway’s streaming HTTP path, while S3RDMA Buffer pays a server\-side staging cost\. Figure[9](https://arxiv.org/html/2605.22850#S4.F9)also shows why transport alone is not sufficient: the single\-object overhead remains visible for small KV chunks\.

### 5\.3\.Layerwise Overlap Model with Hit Rate

What matters for inference is not whether the full prefix arrives quickly, but whether each layer of cached KV arrives before the GPU needs it\. For a context lengthPP\(in tokens\) with prefix hit raterr\(the fraction of thePPcontext tokens that are covered by the cached prefix\), the matched KV bytes per layer areD\(ℓ\)=2​nkv​d​p​\(P​r\)D^\{\(\\ell\)\}=2\\,n\_\{\\mathrm\{kv\}\}\\,d\\,p\\,\(Pr\)\. Given the measured per\-layer compute timet\(ℓ\)t^\{\(\\ell\)\}, perfect overlap requires transfer throughputBreq=D\(ℓ\)/t\(ℓ\)B\_\{\\mathrm\{req\}\}=D^\{\(\\ell\)\}/t^\{\(\\ell\)\}\. These expressions connect the byte layout from Equation[1](https://arxiv.org/html/2605.22850#S2.E1)to the TTFT model in Equation[3](https://arxiv.org/html/2605.22850#S3.E3)\. If ObjectCache’s effective per\-layer throughput exceedsBreqB\_\{\\mathrm\{req\}\}, only the first\-layer transfer latency is visible in the TTFT; otherwise, each layer whose transfer exceeds its compute window adds directly to the total TTFT\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x21.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x22.png)\(a\)GET\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x23.png)\(b\)PUT\.

Figure 10\.Per\-request latency breakdown of S3RDMA Direct\. Storage is backend object I/O, Network is RDMA data\-plane transfer, and Control Plane is S3 frontend request and metadata processing\. For small objects, fixed control\-plane work dominates the remaining latency after RDMA removes TCP data movement\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x24.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x25.png)

Figure 11\.Server\-side aggregation amortizes per\-object overhead and achieves high speedups at small chunk granularities\.
### 5\.4\.Per\-Request Overhead and Overlap Feasibility

Figures[10](https://arxiv.org/html/2605.22850#S5.F10)and[11](https://arxiv.org/html/2605.22850#S5.F11)first isolate the two interface costs that ObjectCache must address: fixed per\-request control cost and low throughput for fine\-grained objects\. After RDMA reduces transfer overhead, HTTP and RGW metadata work dominate the remaining per\-request cost at small objects \(Figure[10](https://arxiv.org/html/2605.22850#S5.F10)\)\. Figure[11](https://arxiv.org/html/2605.22850#S5.F11)shows that batching and server\-side aggregation turn many small object reads into larger layerwise transfers and recover high throughput for KV cache loads\. Together, these measurements show why ObjectCache needs both pieces: batching amortizes fixed request overhead, while aggregation makes fine\-grained KV chunks efficient to transfer\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x26.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x27.png)

\(a\)Llama 3\.1 8B per\-layer compute time\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x28.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x29.png)

\(b\)Llama 3\.1 8B required transfer throughput to overlap\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x30.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x31.png)

\(c\)Granite 3\.3 8B required transfer throughput to overlap\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x32.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x33.png)

\(d\)DeepSeek\-R1\-Distill\-Qwen\-7B required transfer throughput to overlap\.

Figure 12\.A100 layerwise overlap requirements\. Most evaluated configurations require less than 2\.5GB/s of layerwise transfer bandwidth, so ObjectCache can hide cached KV transfer under prefill compute in the dominant regimes\.Figure[12](https://arxiv.org/html/2605.22850#S5.F12)connects aggregation throughput to serving\. The first heatmap reports measured per\-layer compute time for Llama 3\.1 8B, while the remaining heatmaps report the transfer throughput required for Llama, Granite\(IBM,[2026](https://arxiv.org/html/2605.22850#bib.bib40)\), and DeepSeek\(DeepSeek,[2026a](https://arxiv.org/html/2605.22850#bib.bib39)\)models\. Configurations requiring less bandwidth than the ObjectCache layer throughput are compute\-bound; configurations above that boundary suffer from added latency\. The counter\-intuitive takeaway is that longer contexts can relax the transfer budget: although they carry more cached KV bytes, they also create a larger per\-layer compute window in which ObjectCache can hide layerwise transfer\. ObjectCache also reduces many cross\-object range reads per layer to a single RDMA payload per layer \(Appendix Table[A7](https://arxiv.org/html/2605.22850#A6.T7)\)\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x34.png)Figure 13\.TTFT overhead for Llama 3\.1 8B relative to the measured optimal local layerwise baseline for each workload configuration\. CW denotes chunkwise delivery and LW denotes layerwise delivery; S3Agg\-LW stays close to the local baseline except in the transfer\-bound corner where layer delivery cannot be hidden by compute\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x35.png)Figure 14\.Sensitivity of S3\-backed KV loading to bandwidth changes for Llama 3\.1 8B\. Each bar reports the relative TTFT increase when the same path and granularity are capped at 10 Gbps, using its 100 Gbps result as the baseline\.
### 5\.5\.End\-to\-End TTFT

We evaluate end\-to\-end TTFT for Llama 3\.1 8B across two context lengths, 4K and 64K, three prefix hit rates, 12\.5%, 50%, and 87\.5%, and three chunk granularities,G∈\{16,64,256\}G\\in\\\{16,64,256\\\}\. For each workload configuration, the baseline isopt\-local\-LW: a pre\-aggregated KV cache in layer\-major order in pinned host\-memory, so inference requires only host\-to\-device transfers and incurs no runtime aggregation cost\. For all other TTFT measurements, we store KV cache data chunkwise with the specifiedGGin local DRAM or on the remote DAOS server\. Figure[13](https://arxiv.org/html/2605.22850#S5.F13)reports the TTFT overhead of each configuration relative to this baseline\.

The results show that layerwise delivery is important regardless of where the KV cache is stored\. Local\-DRAM\-LW consistently outperforms Local\-DRAM\-CW, indicating that delivering KV blocks in layer order enables better overlap between transfer and GPU computation\. In contrast, S3Batch\-CW incurs the highest overhead in most configurations because it combines chunkwise delivery with S3 control\-plane and network transfer costs\. S3Agg\-LW substantially reduces this overhead and performs close to Local\-DRAM\-LW in most cases\. In several configurations, S3Agg\-LW even achieves lower TTFT than Local\-DRAM\-LW\. We interpret this as an observed resource\-isolation effect: server\-side aggregation uses dedicated CPU cores and parallel SSD reads, whereas Local\-DRAM\-LW consumes client\-side CPU and memory bandwidth that may also be needed by the inference engine\.

The benefit of S3Agg\-LW is most evident for long\-context workloads\. At 64K context length, the per\-layer compute window is sufficiently large to hide most transfer latency, and S3Agg\-LW remains within 0\.1–5\.6% of opt\-local\-LW forG=64G=64across all 64K configurations\. Even at the highest hit rate, where the KV payload reaches 7 GB, the predicted overlap throughput is 3\.1 GB/s \(Appendix Table[A8](https://arxiv.org/html/2605.22850#A7.T8)\), which is below ObjectCache’s sustained aggregation bandwidth\. However,G=16G=16shows noticeably higher overhead thanG=64G=64andG=256G=256, suggesting that small chunk granularity prevents the pipeline from fully exploiting the available server\-side aggregation throughput of approximately 5 GB/s\.

The short\-context 4K workloads are more challenging because the compute window is much smaller\. In these cases, the mainG=64G=64S3Agg\-LW configuration adds 56–75 ms over opt\-local\-LW, and its TTFT can become comparable to S3Batch\-CW\. The limiting case is the 4K, 87\.5% hit\-rate workload, where the baseline compute time is only 64 ms and the required overlap throughput rises to 7\.4 GB/s \(Appendix Table[A8](https://arxiv.org/html/2605.22850#A7.T8)\)\. Although the KV payload is only 448 MB, fixed costs such as RDMA session setup, first\-layer transfer, and control\-plane exchange consume a significant fraction of the available compute window\. As a result, all three granularities converge to similar TTFT overhead in this configuration, indicating that the bottleneck is the short compute window rather than steady\-state transfer throughput\. We therefore classify all three granularities in this configuration as transfer\-bound: layer delivery, not steady\-state bandwidth efficiency, determines the tail\. Chunk granularity has a modest effect on chunkwise methods\. For Local\-DRAM\-CW, largerGGimproves efficiency by reducing chunk management overhead\. For S3Batch\-CW, the effect is weaker because evenG=16G=16produces objects large enough to achieve reasonably high network transfer efficiency\. These 4K results are consistent with the thresholded mode selection in Section[3\.4](https://arxiv.org/html/2605.22850#S3.SS4): small\-payload regimes can be dominated by fixed and startup costs, so ObjectCache should not assume that aggregation is always the better delivery mode\.

### 5\.6\.Sensitivity to Bandwidth Changes

We evaluate bandwidth sensitivity by capping the S3\-backed paths at 10 Gbps and reporting the TTFT increase relative to the corresponding 100 Gbps run\. Figure[14](https://arxiv.org/html/2605.22850#S5.F14)shows that S3Agg\-LW is less sensitive to this bandwidth reduction than S3Batch\-CW except in transfer\-bound configurations\.

This behavior follows from layerwise overlap\. In S3Batch\-CW, computation cannot start until the chunkwise KV load completes, so a bandwidth reduction largely appears as additional TTFT\. In S3Agg\-LW, only the non\-overlapped portion of each layer transfer contributes to TTFT\. Therefore, bandwidth changes affect TTFT mainly when the required overlap throughput exceeds the available bandwidth\. Appendix Table[A8](https://arxiv.org/html/2605.22850#A7.T8)explains where this happens\. At 64K and 50% hit rate, the required throughput is only 0\.50 GB/s, well below 10 Gbps, so S3Agg\-LW shows almost no sensitivity\. At 4K and 50% hit rate, the predicted requirement is 1\.45 GB/s, slightly above 10 Gbps, but the 100 Gbps run is already dominated by fixed costs and pipeline startup overheads; hence the bandwidth cap does not visibly dominate end\-to\-end TTFT\.

The large increases appear at 87\.5% hit rate, where the required throughput exceeds the capped bandwidth: 7\.41 GB/s for 4K and 3\.10 GB/s for 64K\. These configurations become transfer\-bound, as indicated by the markers in Figure[14](https://arxiv.org/html/2605.22850#S5.F14), and therefore show the largest TTFT increase for S3Agg\-LW\. Overall, the results validate that layerwise loading is intrinsically less sensitive to bandwidth changes, but only while per\-layer transfer can be hidden behind compute\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x36.png)Figure 15\.Sensitivity of layerwise TTFT to throttled transfer throughput with S3Agg\-LW\. Each panel normalizes TTFT increase relative to its best measured point\. Dashed lines show the perfect\-overlap bandwidth estimate; dash\-dot lines show the calibrated scheduler target\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x37.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x38.png)\(a\)Workload\-A allocation\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x39.png)\(b\)Workload\-AΔ\\DeltaTTFT\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x40.png)\(c\)Workload\-B allocation\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x41.png)\(d\)Workload\-BΔ\\DeltaTTFT\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x42.png)\(e\)Workload\-C allocation\.
![Refer to caption](https://arxiv.org/html/2605.22850v1/x43.png)\(f\)Workload\-CΔ\\DeltaTTFT\.

Figure 16\.Bandwidth scheduling under shared transfer caps\. For each workload, the left panel shows the bandwidth allocated by each policy and the right panel shows the resulting added TTFT\. Workload\-A uses an 80 Gbps cap; Workload\-B and Workload\-C use 50 Gbps caps\.
### 5\.7\.Bandwidth Allocation for Multi Tenants

We next evaluate how the bandwidth allocator behaves when multiple S3Agg\-LW retrievals share a fixed transfer budget\. Section[3\.6](https://arxiv.org/html/2605.22850#S3.SS6)defines Stall\-opt and Calibrated Stall\-opt; here we compare them against three heuristic bandwidth\-allocation baselines that do not model the layerwise overlap condition\.*Equal*assigns the same bandwidth to every active request, which is attractive for fairness but ignores both KV cache size and compute slack\.*KV\-prop*allocates bandwidth in proportion to the retrieved KV cache size, favoring larger or higher\-hit\-rate requests\.*BW\-prop*allocates bandwidth in proportion to the zero\-stall bandwidth estimate fromBreqB\_\{\\mathrm\{req\}\}, so requests that require higher bandwidth to hide layerwise transfer receive a larger share\. For Calibrated Stall\-opt, we use a 5 Gbps margin, chosen from the S3Agg\-LW rate sweep in Figure[15](https://arxiv.org/html/2605.22850#S5.F15): the dash\-dot calibrated targets shift the analytic perfect\-overlap estimates to the measured knees of the TTFT curves\.

Figure[16](https://arxiv.org/html/2605.22850#S5.F16)compares the scheduling policies on three mixed workloads, and Appendix Table[A9](https://arxiv.org/html/2605.22850#A8.T9)reports the exact per\-request bandwidth allocations\. We construct the workloads from the per\-request bandwidth requirements in Appendix Table[A8](https://arxiv.org/html/2605.22850#A7.T8)\. Workloads A and B contain the same four requests: 16K and 64K contexts at 50% and 87\.5% hit rates\. Together, these requests require 91 Gbps in aggregate, or 111 Gbps after adding the 5 Gbps calibration margin to each request\. The two workloads differ only in their shared bandwidth cap\. Workload A uses an 80 Gbps cap to model moderate contention, where several requests can still remain near their calibrated overlap points\. Workload B uses a 50 Gbps cap to stress the allocator under a tighter budget\. Workload C extends this setting by adding 32K\-context requests, resulting in six tenants with small, medium, and large KV cache footprints under the same 50 Gbps cap\. Its aggregate requirement is 137 Gbps, or 167 Gbps after calibration\. This workload tests whether the scheduler remains effective when the available bandwidth must be distributed across a denser set of transfer sensitivities\. The measured per\-configuration TTFTs are listed in Appendix Tables[A10](https://arxiv.org/html/2605.22850#A8.T10)and[A11](https://arxiv.org/html/2605.22850#A8.T11)\.

Across the three workloads, Calibrated Stall\-opt outperforms the heuristic baselines by allocating bandwidth according to measured overlap requirements rather than per\-request parity, KV size, or analytic demand alone\. Measured as additional TTFT over the effectively unthrottled baseline, it reduces Equal’s penalty by 1\.2–1\.8×\\times\(Figure[16](https://arxiv.org/html/2605.22850#S5.F16)\); Appendix Table[A12](https://arxiv.org/html/2605.22850#A8.T12)gives the full breakdown\. This improvement comes from assigning bandwidth near each request’s useful overlap region\. Equal ignores heterogeneous transfer sensitivity, KV\-prop can starve small but latency\-sensitive requests, and BW\-prop can over\-allocate beyond the point where extra bandwidth reduces TTFT\. Calibrated Stall\-opt avoids these extremes and reduces added TTFT under both moderate and tight caps\. The only exception is Workload C, where Stall\-opt is slightly better because the calibration margin can mildly over\-provision some requests under a dense, heterogeneous 50 Gbps cap\. Even there, Calibrated Stall\-opt remains better than Equal, KV\-prop, and BW\-prop, preserving most of Stall\-opt’s benefit while improving robustness to measurement\-level deviations from the analytic model\.

## 6\.Discussion

### 6\.1\.Position in the KV cache Hierarchy

ObjectCache is not a replacement for hot KV\-cache tier, where GPU memory or DRAM\-backed KV cache pools serve those hottest prefixes\. ObjectCache instead provides a persistent, shared, S3\-compatible capacity tier for large prefix pools, where placement flexibility and low storage cost matter\. The key question is whether it can be made fast enough for serving\-path prefix reuse\. This role complements disaggregated inference systems that separate prefill, decode, and KV cache storage\(Zhonget al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib32); Patelet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib33); Qinet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib1); Huet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib11)\)\. By storing long\-lived prefixes in object storage, ObjectCache lets requests run on any available GPU serving node rather than only where the prefix is cached\.

### 6\.2\.When Layerwise Object Storage Helps

Layerwise ObjectCache is most useful when recomputation is costly, partial prefill computation provides enough per\-layer overlap window, and the storage tier can deliver near the required overlap bandwidth\. When the compute window is too small, bandwidth is below the overlap requirement, or the prefix hit is too small to amortize S3 overheads, the system should instead fall back to chunkwise transfer, a DRAM hot cache, or recomputation\.

### 6\.3\.Limitations and Future Work

In this paper we optimize the serving\-path data movement mechanism for S3\-compatible object storage\. Our prototype evaluates the critical data path and scheduling policy on a small cluster, which is sufficient to isolate the effects of aggregation, RDMA transfer, layerwise delivery, and bandwidth scheduling, but does not capture all production effects\. Future work should validate ObjectCache with concurrent clients, long\-running prefix churn, failures, and integration with batching\(Zhenget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib57)\), chunked prefill\(Agrawalet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib110)\), routing\(Dinget al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib56)\), and eviction policies\(Wanget al\.,[2025a](https://arxiv.org/html/2605.22850#bib.bib7); Fenget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib8)\)\.

ObjectCache also assumes that cached KV can be addressed by prefix chunk and layer range\. Irregular layouts, packed objects, or compressed KV representations\(Changet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib104); Liuet al\.,[2024b](https://arxiv.org/html/2605.22850#bib.bib19)\)are compatible only if the storage server has sufficient metadata to translate layerwise requests\. Finally, production deployments need scalable connection management, per\-tenant admission control, and stronger isolation around RDMA credentials\(Louet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib58); Rothenbergeret al\.,[2021](https://arxiv.org/html/2605.22850#bib.bib76); Tsaiet al\.,[2019](https://arxiv.org/html/2605.22850#bib.bib77); Simpsonet al\.,[2020](https://arxiv.org/html/2605.22850#bib.bib78); Zhaoet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib79)\)\.

## 7\.Related Work

KV cache management and prefix reuse\.Paged KV allocation, radix\-tree prefix matching, and KV offloading systems provide the software substrate that ObjectCache builds on\(Kwonet al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib13); Yeet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib17); Shenget al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib18); Leeet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib20); Chenet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib5); Yaoet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib14); Yuet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib6); Zouet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib4); Prabhuet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib93)\)\. These systems typically store KV in fixed\-size chunks\. ObjectCache preserves this chunkwise, hash\-addressed layout, but changes how remote storage delivers matched chunks to the serving node\. CacheFlow\(Nianet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib102)\)accelerates KV cache restoration by scheduling recomputation and I/O across token, layer, and GPU dimensions inside the serving system\. ObjectCache addresses a different bottleneck: how an S3\-compatible object tier should fetch, aggregate, and deliver many matched KV chunks in layer order to the inference engine\.

Disaggregated LLM serving\.Prior work on prefill/decode separation and remote KV pools shows that LLM serving can benefit from decoupling compute roles and moving KV cache data over the network\(Zhonget al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib32); Patelet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib33); Qinet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib1); Huet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib11),[2025a](https://arxiv.org/html/2605.22850#bib.bib34),[2025b](https://arxiv.org/html/2605.22850#bib.bib12); Qinet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib9); Chenet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib10); Chenget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib82); Fuet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib92); Meiet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib90); Suet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib89); Kamathet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib94)\)\. ObjectCache follows this architectural direction, but targets a persistent S3\-compatible object storage which targets for a cloud native solution\.

S3\-over\-RDMA and object storage\.Recent S3\-over\-RDMA systems split the HTTP control path from an RDMA data path, improving large\-object transfer over TCP\-based S3\(NVIDIA,[2026b](https://arxiv.org/html/2605.22850#bib.bib2),[a](https://arxiv.org/html/2605.22850#bib.bib3); MinIO,[2025](https://arxiv.org/html/2605.22850#bib.bib48); NVIDIA,[2025](https://arxiv.org/html/2605.22850#bib.bib49); VAST Data,[2025](https://arxiv.org/html/2605.22850#bib.bib50); Cloudian,[2025](https://arxiv.org/html/2605.22850#bib.bib51); Dell Technologies,[2026](https://arxiv.org/html/2605.22850#bib.bib52); Sunet al\.,[2025](https://arxiv.org/html/2605.22850#bib.bib81); Haoet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib83)\)\. Low\-latency object stores such as S3 Express reduce access latency but still expose single\-object request semantics\(Zhouet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib66)\)\. ObjectCache builds on this trend by adding KV\-aware multi\-object aggregation and layerwise delivery to S3\-compatible storage\.

KV compression and small\-state attention\.KV quantization, compression, MLA\-style architectures, recent\-window methods, and state\-retention techniques reduce the bytes retained per token\(Shazeer,[2019](https://arxiv.org/html/2605.22850#bib.bib22); Ainslieet al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib23); Liuet al\.,[2024a](https://arxiv.org/html/2605.22850#bib.bib24); Zhanget al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib25); Liuet al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib26); Xiaoet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib27); Liet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib28); Geet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib29); Liuet al\.,[2024c](https://arxiv.org/html/2605.22850#bib.bib30); Hooperet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib31);[Sunet al\.,](https://arxiv.org/html/2605.22850#bib.bib21); Fenget al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib8); Zandiehet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib80); Duet al\.,[2026](https://arxiv.org/html/2605.22850#bib.bib91)\)\. These techniques are complementary to ObjectCache\. They can reduce transfer volume, but they also shrink each useful object payload, making S3 overhead more dominant\.

## 8\.Conclusion

S3\-compatible object storage is a natural storage tier for large prefix KV cache pools, but standard S3 exposes the mismatched serving\-path semantics\. ObjectCache adds the missing serving\-path semantics via protocol\-scheduling co\-design\. Our prototype shows that these pieces must work together\. RDMA improves raw S3 transport, aggregation amortizes network overhead, layerwise delivery overlaps transfer with GPU compute, and bandwidth\-aware scheduling reduces added TTFT under shared bandwidth\. On a 100 Gbps RoCE prototype with Llama 3\.1 8B, ObjectCache reaches within 5\.6% of the optimal local DRAM solution at 64K context length\. Our results suggest that object storage can serve as a runtime KV cache backend rather than only a cold archive, while retaining the persistence, capacity, placement flexibility, and low storage cost that make object storage attractive\.

## References

- M\. Abou Ali, F\. Dornaika, and J\. Charafeddine \(2025\)Agentic ai: a comprehensive survey of architectures, applications, and future directions\.Artificial Intelligence Review59\(1\),pp\. 11\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p1.1)\.
- A\. Agrawal, N\. Kedia, A\. Panwar, J\. Mohan, N\. Kwatra, B\. S\. Gulavani, A\. Tumanov, and R\. Ramjee \(2025\)Efficient llm inference via chunked prefills\.ACM SIGOPS Operating Systems Review59\(1\),pp\. 9–16\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p1.1)\.
- J\. Ainslie, J\. Lee\-Thorp, M\. De Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai \(2023\)Gqa: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Anthropic \(2026\)Models & pricing\.Note:[https://platform\.claude\.com/docs/en/about\-claude/pricing](https://platform.claude.com/docs/en/about-claude/pricing)Cited by:[Figure A2](https://arxiv.org/html/2605.22850#A2.F2),[Figure A2](https://arxiv.org/html/2605.22850#A2.F2.3.2)\.
- Ceph \(2026\)Ceph \- a scalable distributed storage system\.Note:[https://github\.com/ceph/ceph](https://github.com/ceph/ceph)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p6.1),[§4](https://arxiv.org/html/2605.22850#S4.p1.1)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, M\. Abdelfattah, and K\. Wu \(2025\)Palu: kv\-cache compression with low\-rank projection\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 50222–50249\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p2.1),[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- W\. Chen, S\. He, H\. Qu, R\. Zhang, S\. Yang, P\. Chen, Y\. Zheng, B\. Huai, and G\. Chen \(2025\)\{\\\{impress\}\\\}: An\{\\\{importance\-informed\}\\\}\{\\\{multi\-tier\}\\\}prefix\{\\\{kv\}\\\}storage system for large language model inference\.In23rd USENIX Conference on File and Storage Technologies \(FAST 25\),pp\. 187–201\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- Y\. Chen, W\. Cui, H\. Zhao, Z\. Xu, X\. Fan, X\. Chen, Y\. Zhou, S\. Sun, B\. He, and Q\. Chen \(2026\)Towards high\-goodput llm serving with prefill\-decode multiplexing\.InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,pp\. 2030–2047\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- R\. Cheng, Y\. Lai, X\. Wei, R\. Chen, and H\. Chen \(2026\)KUNSERVE: parameter\-centric memory management for efficient memory overloading handling in LLM serving\.InProceedings of the 21st European Conference on Computer Systems,EuroSys ’26,pp\. 1244–1260\.External Links:[Document](https://dx.doi.org/10.1145/3767295.3769348)Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- Cloudian \(2025\)Supercharging vector database indexing: 8x faster with cloudian s3 rdma, milvus and nvidia\.Note:[https://cloudian\.com/blog/supercharging\-vector\-database\-indexing\-8x\-faster\-with\-cloudian\-s3\-rdma\-and\-nvidia/](https://cloudian.com/blog/supercharging-vector-database-indexing-8x-faster-with-cloudian-s3-rdma-and-nvidia/)Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)Flashattention: fast and memory\-efficient exact attention with io\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.6)\.
- DAOS \(2026\)DAOS\.Note:[https://docs\.daos\.io/v2\.6/](https://docs.daos.io/v2.6/)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p6.1),[§4](https://arxiv.org/html/2605.22850#S4.p1.1)\.
- DeepSeek \(2026a\)DeepSeek r1 distill qwen 7b\.Note:[https://huggingface\.co/deepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)Cited by:[§5\.4](https://arxiv.org/html/2605.22850#S5.SS4.p2.1)\.
- DeepSeek \(2026b\)Models & pricing\.Note:[https://api\-docs\.deepseek\.com/quick\_start/pricing](https://api-docs.deepseek.com/quick_start/pricing)Cited by:[Figure A2](https://arxiv.org/html/2605.22850#A2.F2),[Figure A2](https://arxiv.org/html/2605.22850#A2.F2.3.2)\.
- Dell Technologies \(2026\)Accelerating ai workloads with rdma for s3\-compatible storage: a game\-changer with dell objectscale\.Note:[https://infohub\.delltechnologies\.com/en\-uk/p/accelerating\-ai\-workloads\-with\-rdma\-for\-s3\-compatible\-storage\-a\-game\-changer\-with\-dell\-objectscale/](https://infohub.delltechnologies.com/en-uk/p/accelerating-ai-workloads-with-rdma-for-s3-compatible-storage-a-game-changer-with-dell-objectscale/)Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Rühle, L\. Lakshmanan, and A\. H\. Awadallah \(2024\)Hybrid llm: cost\-efficient and quality\-aware query routing\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 41348–41366\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p1.1)\.
- D\. Du, S\. Cao, J\. Cheng, L\. Mai, T\. Cao, and M\. Yang \(2026\)Bitdecoding: unlocking tensor cores for long\-context llms with low\-bit kv cache\.In2026 IEEE International Symposium on High Performance Computer Architecture \(HPCA\),pp\. 1–13\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- W\. Fan, Y\. Ding, L\. Ning, S\. Wang, H\. Li, D\. Yin, T\. Chua, and Q\. Li \(2024\)A survey on rag meeting llms: towards retrieval\-augmented large language models\.InProceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining,pp\. 6491–6501\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p1.1)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2026\)Ada\-kv: optimizing kv cache eviction by adaptive budget allocation for efficient llm inference\.Advances in Neural Information Processing Systems38,pp\. 113152–113188\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p1.1),[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Y\. Fu, L\. Xue, Y\. Huang, A\. Brabete, D\. Ustiugov, Y\. Patel, and L\. Mai \(2024\)\{\\\{serverlessllm\}\\\}:\{\\\{low\-Latency\}\\\}serverless inference for large language models\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 135–153\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- B\. Gao, Z\. He, P\. Sharma, Q\. Kang, D\. Jevdjic, J\. Deng, X\. Yang, Z\. Yu, and P\. Zuo \(2024\)\{\\\{cost\-Efficient\}\\\}large language model serving for multi\-turn conversations with\{\\\{cachedattention\}\\\}\.In2024 USENIX annual technical conference \(USENIX ATC 24\),pp\. 111–126\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.8)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024\)Model tells you what to discard: adaptive kv cache compression for llms\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 22975–22988\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. Zhong \(2024\)Prompt cache: modular attention reuse for low\-latency inference\.Proceedings of Machine Learning and Systems6,pp\. 325–338\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.8)\.
- Google \(2026\)Models & pricing\.Note:[https://cloud\.google\.com/gemini\-enterprise\-agent\-platform/generative\-ai/pricing](https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing)Cited by:[Figure A2](https://arxiv.org/html/2605.22850#A2.F2),[Figure A2](https://arxiv.org/html/2605.22850#A2.F2.3.2)\.
- Y\. Hao, T\. Yao, X\. Wei, D\. Zhang, T\. Sun, Y\. Zhang, Z\. Fu, H\. Wu, and R\. Chen \(2026\)Fast cloud storage for\{\\\{ai\}\\\}jobs via grouped\{\\\{i/o\}\\\}\{\\\{api\}\\\}with transparent\{\\\{read/write\}\\\}optimizations\.In24th USENIX Conference on File and Storage Technologies \(FAST 26\),pp\. 255–270\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)Kvquant: towards 10 million context length llm inference with kv cache quantization\.Advances in Neural Information Processing Systems37,pp\. 1270–1303\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- C\. Hu, H\. Huang, J\. Hu, J\. Xu, X\. Chen, T\. Xie, C\. Wang, S\. Wang, Y\. Bao, N\. Sun,et al\.\(2024\)Memserve: context caching for disaggregated llm serving with elastic memory pool\.arXiv preprint arXiv:2406\.17565\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p2.1),[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.2.1.1),[§6\.1](https://arxiv.org/html/2605.22850#S6.SS1.p1.1),[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- C\. Hu, H\. Huang, L\. Xu, X\. Chen, C\. Wang, J\. Xu, S\. Chen, H\. Feng, S\. Wang, Y\. Bao,et al\.\(2025a\)ShuffleInfer: disaggregate llm inference for mixed downstream workloads\.ACM Transactions on Architecture and Code Optimization22\(2\),pp\. 1–24\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- J\. Hu, J\. Xu, Z\. Liu, Y\. He, Y\. Chen, H\. Xu, J\. Liu, J\. Meng, B\. Zhang, S\. Wan,et al\.\(2025b\)\{\\\{deepserve\}\\\}: Serverless large language model serving at scale\.In2025 USENIX Annual Technical Conference \(USENIX ATC 25\),pp\. 57–72\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- IBM \(2026\)Granite 3\.3 8b\.Note:[https://huggingface\.co/ibm\-granite/granite\-3\.3\-8b\-instruct](https://huggingface.co/ibm-granite/granite-3.3-8b-instruct)Cited by:[§5\.4](https://arxiv.org/html/2605.22850#S5.SS4.p2.1)\.
- A\. K\. Kamath, R\. Prabhu, J\. Mohan, S\. Peter, R\. Ramjee, and A\. Panwar \(2025\)Pod\-attention: unlocking full prefill\-decode overlap for faster llm inference\.InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2,pp\. 897–912\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Table A6](https://arxiv.org/html/2605.22850#A6.T6),[Table A6](https://arxiv.org/html/2605.22850#A6.T6.2.1),[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.8),[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- W\. Lee, J\. Lee, J\. Seo, and J\. Sim \(2024\)\{\\\{infinigen\}\\\}: Efficient generative inference of large language models with dynamic\{\\\{kv\}\\\}cache management\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 155–172\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo,et al\.\(2024a\)Deepseek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p2.1),[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Y\. Liu, Y\. Cheng, J\. Yao, Y\. An, X\. Chen, S\. Feng, Y\. Huang, S\. Shen, R\. Zhang, K\. Du,et al\.\(2025\)Lmcache: an efficient kv cache layer for enterprise\-scale llm inference\.arXiv preprint arXiv:2510\.09665\.Cited by:[§3\.3](https://arxiv.org/html/2605.22850#S3.SS3.p1.2)\.
- Y\. Liu, H\. Li, Y\. Cheng, S\. Ray, Y\. Huang, Q\. Zhang, K\. Du, J\. Yao, S\. Lu, G\. Ananthanarayanan,et al\.\(2024b\)Cachegen: kv cache compression and streaming for fast large language model serving\.InProceedings of the ACM SIGCOMM 2024 Conference,pp\. 38–56\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.Advances in Neural Information Processing Systems36,pp\. 52342–52364\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024c\)KIVI: a tuning\-free asymmetric 2bit quantization for kv cache\.InProceedings of the 41st International Conference on Machine Learning,pp\. 32332–32344\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- LMCache \(2026\)LMCache\.Note:[https://github\.com/lmcache/lmcache](https://github.com/lmcache/lmcache)Cited by:[§4\.2](https://arxiv.org/html/2605.22850#S4.SS2.p1.3)\.
- J\. Lou, X\. Kong, J\. Huang, W\. Bai, N\. S\. Kim, and D\. Zhuo \(2024\)Harmonic: hardware\-assisted\{\\\{rdma\}\\\}performance isolation for public clouds\.In21st USENIX Symposium on Networked Systems Design and Implementation \(NSDI 24\),pp\. 1479–1496\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- Y\. Mei, Y\. Zhuang, X\. Miao, J\. Yang, Z\. Jia, and R\. Vinayak \(2025\)Helix: serving large language models over heterogeneous gpus and network via max\-flow\.InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1,pp\. 586–602\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- F\. Meng, P\. Tang, Z\. Yao, X\. Sun, and M\. Zhang \(2026\)TransMLA: migrating gqa models to mla with full deepseek compatibility and speedup\.Advances in Neural Information Processing Systems38,pp\. 81977–82019\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p2.1)\.
- Meta \(2026\)Llama 3\.1 8b\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.1\-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p6.1),[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p2.1)\.
- MinIO \(2025\)MinIO s3 over rdma\.Note:[https://blog\.min\.io/s3\-over\-rdma/](https://blog.min.io/s3-over-rdma/)Cited by:[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.3.2.1),[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- S\. Nian, J\. Fang, Q\. Feng, Z\. Wu, and F\. Lai \(2026\)CacheFlow: efficient llm serving with 3d\-parallel kv cache restoration\.arXiv preprint arXiv:2604\.25080\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- NVIDIA \(2025\)How to unlock accelerated ai storage performance with rdma for s3\-compatible storage\.Note:[https://blogs\.nvidia\.com/blog/s3\-compatible\-ai\-storage/](https://blogs.nvidia.com/blog/s3-compatible-ai-storage/)Cited by:[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.3.2.1),[§4\.3](https://arxiv.org/html/2605.22850#S4.SS3.p2.1),[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- NVIDIA \(2026a\)NVIDIA cuobject server v1\.0\.0 release notes\.Note:[https://docs\.nvidia\.com/gpudirect\-storage/cuobject/cuobject\-server\-release\-notes/index\.html](https://docs.nvidia.com/gpudirect-storage/cuobject/cuobject-server-release-notes/index.html)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p4.1),[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.3.2.1),[§4\.3](https://arxiv.org/html/2605.22850#S4.SS3.p2.1),[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- NVIDIA \(2026b\)NVIDIA cuobject: gpudirect storage for objects\.Note:[https://docs\.nvidia\.com/gpudirect\-storage/cuobject/index\.html](https://docs.nvidia.com/gpudirect-storage/cuobject/index.html)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p4.1),[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.3.2.1),[§4\.3](https://arxiv.org/html/2605.22850#S4.SS3.p2.1),[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- Nvidia \(2026\)NVIDIA inference xfer library \(nixl\)\.Note:[https://github\.com/ai\-dynamo/nixl](https://github.com/ai-dynamo/nixl)Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p6.1),[§4](https://arxiv.org/html/2605.22850#S4.p1.1)\.
- OpenAI \(2026\)Models & pricing\.Note:[https://developers\.openai\.com/api/docs/pricing](https://developers.openai.com/api/docs/pricing)Cited by:[Figure A2](https://arxiv.org/html/2605.22850#A2.F2),[Figure A2](https://arxiv.org/html/2605.22850#A2.F2.3.2)\.
- P\. Patel, E\. Choukse, C\. Zhang, A\. Shah, Í\. Goiri, S\. Maleki, and R\. Bianchini \(2024\)Splitwise: efficient generative llm inference using phase splitting\.In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture \(ISCA\),pp\. 118–132\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p3.1),[§6\.1](https://arxiv.org/html/2605.22850#S6.SS1.p1.1),[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- R\. Prabhu, A\. Nayak, J\. Mohan, R\. Ramjee, and A\. Panwar \(2025\)Vattention: dynamic memory management for serving llms without pagedattention\.InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1,pp\. 1133–1150\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- R\. Qin, W\. He, Y\. Wang, Z\. Li, X\. Xu, Y\. Wu, W\. Zheng, and M\. Zhang \(2026\)Prefill\-as\-a\-service: kvcache of next\-generation models could go cross\-datacenter\.arXiv preprint arXiv:2604\.15039\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- R\. Qin, Z\. Li, W\. He, J\. Cui, H\. Tang, F\. Ren, T\. Ma, S\. Cai, Y\. Zhang, M\. Zhang,et al\.\(2024\)Mooncake: a kvcache\-centric disaggregated architecture for llm serving\.ACM Transactions on Storage\.Cited by:[Table A6](https://arxiv.org/html/2605.22850#A6.T6),[Table A6](https://arxiv.org/html/2605.22850#A6.T6.2.1),[§1](https://arxiv.org/html/2605.22850#S1.p2.1),[§3\.3](https://arxiv.org/html/2605.22850#S3.SS3.p1.2),[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.2.1.1),[§6\.1](https://arxiv.org/html/2605.22850#S6.SS1.p1.1),[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- B\. Rothenberger, K\. Taranov, A\. Perrig, and T\. Hoefler \(2021\)\{\\\{redmark\}\\\}: Bypassing\{\\\{rdma\}\\\}security mechanisms\.In30th USENIX Security Symposium \(USENIX Security 21\),pp\. 4277–4292\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- N\. Shazeer \(2019\)Fast transformer decoding: one write\-head is all you need\.arXiv preprint arXiv:1911\.02150\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Ré, I\. Stoica, and C\. Zhang \(2023\)Flexgen: high\-throughput generative inference of large language models with a single gpu\.InInternational Conference on Machine Learning,pp\. 31094–31116\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- A\. K\. Simpson, A\. Szekeres, J\. Nelson, and I\. Zhang \(2020\)Securing\{\\\{rdma\}\\\}for\{\\\{high\-performance\}\\\}datacenter storage systems\.In12th USENIX Workshop on Hot Topics in Cloud Computing \(HotCloud 20\),Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- Z\. Su, Z\. Zhang, T\. Lan, Z\. Wang, J\. Yang, and Y\. Cheng \(2026\)MorphServe: efficient and workload\-aware llm serving via runtime quantized layer swapping and kv cache resizing\.InProceedings of Machine Learning and Systems,Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- \[61\]H\. Sun, L\. Chang, W\. Bao, S\. Zheng, N\. Zheng, X\. Liu, H\. Dong, Y\. Chi, and B\. ChenShadowKV: kv cache in shadows for high\-throughput long\-context llm inference\.InForty\-second International Conference on Machine Learning,Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- S\. Sun, I\. Khor, J\. Shin, and P\. Desnoyers \(2025\)A fast, efficient, and strongly\-consistent object store\.InProceedings of the 2025 ACM Symposium on Cloud Computing,pp\. 708–721\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- S\. Tsai, M\. Payer, and Y\. Zhang \(2019\)Pythia: remote oracles for the masses\.In28th USENIX Security Symposium \(USENIX Security 19\),pp\. 693–710\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- UCX \(2026\)Unified communication x\.Note:[https://github\.com/openucx/ucx](https://github.com/openucx/ucx)Cited by:[§4\.5](https://arxiv.org/html/2605.22850#S4.SS5.p1.1)\.
- VAST Data \(2025\)S3 over rdma: scaling the kv cache data plane\.Note:[https://www\.vastdata\.com/blog/s3\-over\-rdma\-scaling\-the\-kv\-cache\-data\-plane](https://www.vastdata.com/blog/s3-over-rdma-scaling-the-kv-cache-data-plane)Cited by:[Table 2](https://arxiv.org/html/2605.22850#S3.T2.4.3.2.1),[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.6)\.
- vLLM \(2026\)VLLM\.Note:[https://github\.com/vllm\-project/vllm](https://github.com/vllm-project/vllm)Cited by:[§4\.2](https://arxiv.org/html/2605.22850#S4.SS2.p1.3)\.
- J\. Wang, J\. Han, X\. Wei, S\. Shen, D\. Zhang, C\. Fang, R\. Chen, W\. Yu, and H\. Chen \(2025a\)\{\\\{kvcache\}\\\}Cache in the wild: characterizing and optimizing\{\\\{kvcache\}\\\}cache at a large cloud provider\.In2025 USENIX Annual Technical Conference \(USENIX ATC 25\),pp\. 465–482\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.8),[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p1.1)\.
- Y\. Wang, Y\. Chen, Z\. Li, X\. Kang, Y\. Fang, Y\. Zhou, Y\. Zheng, Z\. Tang, X\. He, R\. Guo,et al\.\(2025b\)Burstgpt: a real\-world workload dataset to optimize llm serving systems\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5831–5841\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p1.1)\.
- Y\. Xiang, X\. Li, K\. Qian, Y\. Zhang, W\. Yu, E\. Zhai, X\. Jin, and J\. Zhou \(2026\)\{\\\{servegen\}\\\}: Workload characterization and generation of large language model serving in production\.In23rd USENIX Symposium on Networked Systems Design and Implementation \(NSDI 26\),pp\. 1845–1859\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21875–21895\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- J\. Yao, H\. Li, Y\. Liu, S\. Ray, Y\. Cheng, Q\. Zhang, K\. Du, S\. Lu, and J\. Jiang \(2025\)Cacheblend: fast large language model serving for rag with cached knowledge fusion\.InProceedings of the twentieth European conference on computer systems,pp\. 94–109\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- L\. Ye, Z\. Tao, Y\. Huang, and Y\. Li \(2024\)Chunkattention: efficient self\-attention with prefix\-aware kv cache and two\-phase partition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11608–11620\.Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p1.8),[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- W\. Yu, Y\. Jiarui, H\. Mengke, J\. Liu, and S\. Luo \(2025\)On 10x better scalability: kv stores scale up kv cache\.arXiv preprint arXiv:2511\.16138\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.
- A\. Zandieh, M\. Daliri, M\. Hadian, and V\. Mirrokni \(2026\)TurboQuant: online vector quantization with near\-optimal distortion rate\.InProceedings of the International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2605.22850#S2.SS1.p2.1),[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p4.1)\.
- C\. Zhao, J\. Min, M\. Liu, and A\. Krishnamurthy \(2025\)\{\\\{white\-Boxing\}\\\}\{\\\{rdma\}\\\}with\{\\\{packet\-granular\}\\\}software control\.In22nd USENIX Symposium on Networked Systems Design and Implementation \(NSDI 25\),pp\. 427–449\.Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p2.1)\.
- Z\. Zheng, X\. Ji, T\. Fang, F\. Zhou, C\. Liu, and G\. Peng \(2026\)BatchLLM: optimizing large batched llm inference with global prefix sharing and throughput\-oriented token batching\.InProceedings of Machine Learning and Systems \(MLSys\),Cited by:[§6\.3](https://arxiv.org/html/2605.22850#S6.SS3.p1.1)\.
- Y\. Zhong, S\. Liu, J\. Chen, J\. Hu, Y\. Zhu, X\. Liu, X\. Jin, and H\. Zhang \(2024\)\{\\\{distserve\}\\\}: Disaggregating prefill and decoding for goodput\-optimized large language model serving\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 193–210\.Cited by:[§1](https://arxiv.org/html/2605.22850#S1.p3.1),[§6\.1](https://arxiv.org/html/2605.22850#S6.SS1.p1.1),[§7](https://arxiv.org/html/2605.22850#S7.p2.1)\.
- J\. Zhou, K\. Huang, and T\. Wang \(2026\)Milliscale: fast commit on low\-latency object storage\.arXiv preprint arXiv:2603\.02108\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p3.1)\.
- J\. Zou, S\. Wu, H\. Duan, Q\. Li, and C\. J\. Xue \(2026\)ContiguousKV: accelerating llm prefill with granularity\-aligned kv cache management\.arXiv preprint arXiv:2601\.13631\.Cited by:[§7](https://arxiv.org/html/2605.22850#S7.p1.1)\.

## Appendix AAdditional Motivation Data

Tables[A1](https://arxiv.org/html/2605.22850#A1.T1)and[A2](https://arxiv.org/html/2605.22850#A1.T2)report the full steady\-state measurements behind the cache\-boundary granularity study\. For each semantic hit boundary, the prompt reusesM−GM\-Gtokens, whereM=C⋅rM=C\\cdot ris the base hit boundary andG∈\{16,512\}G\\in\\\{16,512\\\}\. ThusG=512G=512recomputes 496 more tokens thanG=16G=16in every row\. Entries are TTFT in milliseconds over steady\-state trials 1–2, reported as mean±\\pmone standard deviation\.Δ=TG=512−TG=16\\Delta=T\_\{G=512\}\-T\_\{G=16\}uses the same steady\-state trials and excludes trial 0 to avoid shape\-warmup effects\.

Table A1\.Steady\-state boundary\-granularity recompute cost for Llama 3\.1 8B\.Table A2\.Steady\-state boundary\-granularity recompute cost for DeepSeek\-R1\-Distill\-Qwen\-7B\.
## Appendix BMeasured Cache\-Boundary Recompute

![Refer to caption](https://arxiv.org/html/2605.22850v1/x44.png)Figure A1\.Repository\-scale prefixes can translate into large KV cache footprints\. Ranges are computed from repository token counts and two representative KV cache sizes per token\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x45.png)Figure A2\.Major APIs price cached input far below fresh input, reflecting the economic value of prefix reuse in deployed LLM services\(DeepSeek,[2026b](https://arxiv.org/html/2605.22850#bib.bib72); Anthropic,[2026](https://arxiv.org/html/2605.22850#bib.bib74); Google,[2026](https://arxiv.org/html/2605.22850#bib.bib75); OpenAI,[2026](https://arxiv.org/html/2605.22850#bib.bib73)\)\.
## Appendix CPinned CPU\-GPU H2D Microbenchmarks

![Refer to caption](https://arxiv.org/html/2605.22850v1/x46.png)Figure A3\.Pinned CPU→\\toGPU H2D throughput and latency vs\. block size for A100 \(PCIe Gen4 x8\) and H100 \(PCIe Gen5 x16\)\. Bars report mean throughput \(±\\pm1σ\\sigmaover 200 per\-size samples\); right\-axis lines report P50/P99 latency across 64 KB–64 MB blocks\. The source buffer is placed on NUMA 0 in both setups\. A100 saturates near 12 GB/s; H100 saturates near 55 GB/s for blocks≥\\geq8 MB\.
## Appendix DPer\-model, per\-GPU recompute and bandwidth heatmaps

Figures[A4](https://arxiv.org/html/2605.22850#A4.F4)–[A7](https://arxiv.org/html/2605.22850#A4.F7)extend the main\-paper recompute and required\-throughput surfaces across the four representative 7–8B models and both GPU platforms\. Total cached KV size is GPU\-agnostic \(Figure[A4](https://arxiv.org/html/2605.22850#A4.F4)\); per\-layer compute time \(Figure[A5](https://arxiv.org/html/2605.22850#A4.F5)\), MFU \(Figure[A6](https://arxiv.org/html/2605.22850#A4.F6)\), and required bandwidth \(Figure[A7](https://arxiv.org/html/2605.22850#A4.F7)\) each stack A100 \(top row\) and H100 \(bottom row\) to expose the GPU\-platform effect at constant model and context\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x47.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x48.png)\(a\)Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x49.png)\(b\)Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x50.png)\(c\)DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x51.png)\(d\)Granite 3\.3 8B

Figure A4\.Total cached KV size per \(context length, hit rate\) configuration for each model\. KV size is independent of GPU platform, so each value is reported once\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x52.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x53.png)\(a\)A100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x54.png)\(b\)A100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x55.png)\(c\)A100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x56.png)\(d\)A100, Granite 3\.3 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x57.png)\(e\)H100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x58.png)\(f\)H100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x59.png)\(g\)H100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x60.png)\(h\)H100, Granite 3\.3 8B

Figure A5\.Per\-layer compute time across four representative models on A100 \(top row\) and H100 \(bottom row\)\. Blank configurations indicate unsupported or unmeasured context lengths, such as Mistral 7B beyond its 32K window\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x61.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x62.png)\(a\)A100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x63.png)\(b\)A100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x64.png)\(c\)A100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x65.png)\(d\)A100, Granite 3\.3 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x66.png)\(e\)H100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x67.png)\(f\)H100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x68.png)\(g\)H100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x69.png)\(h\)H100, Granite 3\.3 8B

Figure A6\.Model FLOP utilization \(MFU\) across four representative models on A100 \(top row\) and H100 \(bottom row\)\. Low\-MFU cells indicate configurations that do not saturate GPU compute; high\-MFU cells identify compute\-dense configurations\.![Refer to caption](https://arxiv.org/html/2605.22850v1/x70.png)

![Refer to caption](https://arxiv.org/html/2605.22850v1/x71.png)\(a\)A100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x72.png)\(b\)A100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x73.png)\(c\)A100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x74.png)\(d\)A100, Granite 3\.3 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x75.png)\(e\)H100, Llama 3\.1 8B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x76.png)\(f\)H100, Mistral 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x77.png)\(g\)H100, DeepSeek\-Distill 7B
![Refer to caption](https://arxiv.org/html/2605.22850v1/x78.png)\(h\)H100, Granite 3\.3 8B

Figure A7\.Required per\-layer transfer throughput to fully overlap cached KV loading with compute across four models on A100 \(top row\) and H100 \(bottom row\)\. Faster compute on H100 raises the bandwidth needed for perfect overlap\.
## Appendix EOptimal Aggregation Size Benchmark

Here we test what is the optimal aggregation size for different chunk granularities\.

![Refer to caption](https://arxiv.org/html/2605.22850v1/x79.png)Figure A8\.S3RDMA Agg throughput across aggregation sizes and chunk granularities\. The sweep coversagg\_size∈\\in\{512KB,1MB,2MB,4MB,8MB\} andG∈\{16,32,64,128,256,512\}G\\in\\\{16,32,64,128,256,512\\\}\. Throughput peaks atG=256G\{=\}256withagg\_size=2 MB \(9\.98 GB/s\); forG=16G\{=\}16,agg\_size=1 MB is best\.
## Appendix FReference Algorithms and Cost Table

This appendix collects the full pseudocode for the ObjectCache layerwise GET and the Calibrated Stall\-opt scheduler, together with the storage\-tier cost breakdown referenced in the main discussion\.

Table A3\.ObjectCache layerwise GET \(full pseudocode\)\.Table A4\.Calibrated Stall\-opt layerwise bandwidth allocation \(full pseudocode\)\.Table A5\.Approximate cost comparison across storage tiers \(2025 cloud pricing\)\.Table A6\.Extra GPU prefill latency per cache\-hit boundary when chunk granularity increases from G=16 \(vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.22850#bib.bib13)\)default\) to G=512 \(Mooncake\(Qinet al\.,[2024](https://arxiv.org/html/2605.22850#bib.bib1)\)\), measured on A100/H100\. Each boundary recomputes up to 496 redundant tokens\. At 1K QPS, the Llama 8B\-64K penalty wastes∼\\sim1,500 GPU hours per day for both A100 and H100\.- •Δ\\DeltaFLOPs: extra recompute work per request from the coarser chunk tail \(496 additional tokens\), constant across hit ratesr∈\{12\.5%,25%,…,87\.5%\}r\\in\\\{12\.5\\%,25\\%,\\ldots,87\.5\\%\\\}\. Latency rows: mean±\\pmstd ofΔ​t=TG=512−TG=16\\Delta t=T\_\{G\{=\}512\}\-T\_\{G\{=\}16\}measured at eachrrwith one chunk miss\. Steady\-state after warm\-up; full per\-hit breakdowns in Appendix Tables 1–2\.

Table A7\.Client\-visible element count for S3RDMA Agg bounded layerwise aggregation\.
## Appendix GCompute time and required throughput at the canonical configurations

Table[A8](https://arxiv.org/html/2605.22850#A7.T8)reports total prefill compute time, per\-layer compute time, and the per\-layer transfer throughput needed for full overlap, for the\(C,r\)\(C,r\)configurations used throughout the streaming\-overlap analysis and scheduler workloads\.

Table A8\.Per\-layer compute time and required transfer throughput to fully overlap one layer of cached KV loading with compute, for Llama 3\.1 8B on A100 80GB\. Total cached KV bytes =r⋅C⋅L⋅br\\cdot C\\cdot L\\cdot\\mathrm\{b\}withL=32L\{=\}32layers andb=4096\\mathrm\{b\}\{=\}4096bytes per token per layer\.Tcompute\(ℓ\)=Ttotal/LT^\{\(\\ell\)\}\_\{\\mathrm\{compute\}\}=T\_\{\\mathrm\{total\}\}/L\. Required transfer throughput=KV/Ttotal=\\mathrm\{KV\}/T\_\{\\mathrm\{total\}\}\(equivalently per\-layer\-KV / per\-layer\-compute\)\.
## Appendix HPer\-request allocations behind Figure[16](https://arxiv.org/html/2605.22850#S5.F16)

Table[A9](https://arxiv.org/html/2605.22850#A8.T9)reports the per\-request transfer throughput each scheduling policy assigns under shared 80 Gbps and 50 Gbps caps to show moderate and stronger contention for the scheduler workloads\.

Table A9\.Per\-request allocated transfer throughput \(Gbps\) for the parallel scheduler workloads, using an 80 Gbps cap for Workload A and a 50 Gbps cap for Workloads B and C\.Table[A10](https://arxiv.org/html/2605.22850#A8.T10)reports the measured S3Agg\-LW\-cap TTFT for the exact allocation configurations in Table[A9](https://arxiv.org/html/2605.22850#A8.T9)\. Table[A12](https://arxiv.org/html/2605.22850#A8.T12)sums those per\-request measurements into the total TTFT used by the three scheduling workloads\.

Table A10\.Measured S3Agg\-LW\-cap TTFT for the exact allocation configurations in Table[A9](https://arxiv.org/html/2605.22850#A8.T9)\.Table A11\.Measured S3Agg\-LW\-cap TTFT for the exact allocation configurations in Table[A9](https://arxiv.org/html/2605.22850#A8.T9)\.Table A12\.Total TTFT for the scheduling workloads using the measured S3Agg\-LW\-cap configurations in Tables[A10](https://arxiv.org/html/2605.22850#A8.T10)and[A11](https://arxiv.org/html/2605.22850#A8.T11)\. The baseline is the sum of the same request configurations without an effective rate limit\.