Cartridges at Scale: Training Modular KV Caches over Large Document Collections

# Training Modular KV Caches over Large Document Collections
Source: [https://arxiv.org/html/2606.04557](https://arxiv.org/html/2606.04557)
Momchil Hardalov Gonzalo Iglesias Adrià de Gispert Amazon AGI \{momchilh, gjii, agispert\}@amazon\.com

###### Abstract

Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries\. Cartridges\(Eyubogluet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib8)\)address this by distilling document collections into reusable key\-value \(KV\) caches that eliminate prefilling while preserving accuracy\. A critical limitation of this approach is that cartridges are monolithic and non\-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance\. We introduce*Cartridges at Scale*\(CAS\), a training framework for scalable multi\-cartridge learning with dynamic distractor mixing and a memory\-efficient budget manager that rotates hundreds of per\-document cartridges between GPU and persistent storage\. Our approach scales to collections exceeding a million tokens, improving over a monolithic cartridge by 10–31 points at comparable token budgets\. Oracle cartridge accuracy falls within 2–6 points of full in\-context learning even at high compression\. When paired with retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3–4×\\timesfewer prompt tokens\.

Cartridges at Scale: Training Modular KV Caches over Large Document Collections

Momchil Hardalov Gonzalo Iglesias Adrià de GispertAmazon AGI\{momchilh, gjii, agispert\}@amazon\.com

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.04557v1/x1.png)Figure 1:Performance of per\-document trained Cartridges on LongHealth \(Adamset al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib14)\), docs: 20 patient notes,∼\\sim232K tokens\)\.*Oracle*– only the patient Cartridge is presented in the prompt;*Full*– all cartridges are presented\. Details in §[4](https://arxiv.org/html/2606.04557#S4),*Training Regime*\.Large language models \(LLMs\) can reason over information in their context window\(Snellet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib24); OpenAI,[2024](https://arxiv.org/html/2606.04557#bib.bib27),[2025](https://arxiv.org/html/2606.04557#bib.bib28); DeepSeek\-AI,[2025](https://arxiv.org/html/2606.04557#bib.bib25); Comaniciet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib29); Anthropic,[2025](https://arxiv.org/html/2606.04557#bib.bib30); Agarwalet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib23)\), however this capability comes at a cost that scales with context length—both in latency \(prefill time\) and memory \(KV cache size\)\. For applications requiring persistent access to document collections—such as enterprise knowledge bases, patient records, and legal corpora—repeatedly processing the same documents for each query is prohibitively expensive\.

Retrieval\-augmented generation \(RAG,Lewiset al\.\([2020](https://arxiv.org/html/2606.04557#bib.bib31)\); Guuet al\.\([2020](https://arxiv.org/html/2606.04557#bib.bib70)\)\) mitigates this by retrieving only the most relevant text chunks at query time, but each retrieved chunk is a*fragment*of the original document, forcing the model to reason over incomplete information\.

Prior work has proposed reducing KV\-cache size via token eviction or mergingZhanget al\.\([2023](https://arxiv.org/html/2606.04557#bib.bib34)\); Kanget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib37)\); Liuet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib35)\), or compressing text into softMuet al\.\([2023](https://arxiv.org/html/2606.04557#bib.bib64)\); Chenget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib32)\); Louiset al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib33)\)or discreteJianget al\.\([2023](https://arxiv.org/html/2606.04557#bib.bib38)\); Panet al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib39)\)token representations\. These approaches often require expensive runtime prefills, rely on a separate encoder, or lose information from long documentsŁajewskaet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib40)\)\.

More recently, researchers have explored encoding documents into compact, reusable representations that can be loaded into the model KV cache at inference time without additional computational costDevotoet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib42)\); Kimet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib43)\); Zweigeret al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib44)\)\.*Cartridges*Eyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\)proposed training a small set of key\-value vectors per document collection via context distillation, achieving1010–100×100\\timescompression while preserving the model’s ability to answer questions about the encoded content\. However, their approach relies on a single monolithic cartridge per collection, which does not scale to real\-world document collections: loading all documents into one cache bloats the prefix with irrelevant tokens, while information\-dense content \(e\.g\., financial tables, clinical notes, technical specifications\) cannot be highly compressed without loss\. Moreover, a single monolithic cartridge is inflexible to future changes: updating or adding a single document requires re\-encoding the entire KV cache\. The natural alternative—one cartridge per document—exposes a more fundamental flaw: as Figure[1](https://arxiv.org/html/2606.04557#S1.F1)shows,*mixing cartridges trained in isolation leads to model collapse*—each cartridge performs well in theoracle setting\(one correct cartridge loaded\), but loadingall cartridgessimultaneously drops performance to near\-chance, as the model never learned to selectively attend across independently trained KV prefixes\.

In this work, we introduce CAS, a framework for training and deploying cartridges at scale over collections of hundreds of documents and millions of tokens\. CAS proposes a training regime based on dynamic cartridgerotation and mixingthat closes the performance gap of isolated training \(Fig\.[1](https://arxiv.org/html/2606.04557#S1.F1)\) while remaining feasible within current GPU memory constraints\. We further show that CAS integrates naturally with dense retrieval for cartridge selection, enabling a practical deployment path that matches or exceeds text RAG efficiency\. Our contributions are as follows:

- •We show that independently trained cartridges suffer catastrophic interference at inference time, collapsing to near\-chance performance, and that joint training with distractor cartridges removes this degradation\.
- •We propose CAS, a scalable training framework with GPU↔\\leftrightarrowpersistent storage swapping, optimizer offloading, and prioritized rotation, enabling joint training of hundreds of cartridges within a fixed GPU memory budget\.
- •We propose improved self\-study data generation via proportional\-to\-length sampling and batched multi\-question generation, reducing synthesis cost by up to20×20\\timeswhile improving fact coverage\.
- •We demonstrate that per\-document cartridges consistently outperform a single monolithic cartridge by up to3030points across five diverse benchmarks, with compression tolerance ranging from near\-lossless at100×100\\timesfor short technical documents to up to 5% relative performance drop for table\-heavy financial filings\.
- •We show that Cartridge RAG matches or exceeds Text RAG accuracy at up to4×4\\timesfewer prompt tokens, combining the efficiency of compressed KV caches with the selectivity of dense retrieval\.

![Refer to caption](https://arxiv.org/html/2606.04557v1/images/synth.png)\(a\)Self\-Study\.For each documentdid\_\{i\}, we sample a list of questions usingMQM\_\{Q\}\. Subsequently, the answers along with the logits from the target teacherMAM\_\{A\}are collected\.
![Refer to caption](https://arxiv.org/html/2606.04557v1/images/trainer.png)\(b\)Training\.Each documentdid\_\{i\}is assigned a dedicated cartridge\. A*budget manager*keeps onlyBBcartridges on GPU and rotates the pool everyRRsteps\.
![Refer to caption](https://arxiv.org/html/2606.04557v1/images/inference.png)\(c\)Inference\.A selection function activateskkcartridges from the store\. The selected cartridges are concatenated and prepended to the query tokens before decoding\.

Figure 2:Cartridges at Scale \(CAS\) End\-to\-End Pipeline\.
## 2Cartridges at Scale \(CAS\) Training

#### Background\.

Our work builds on the Cartridge framework ofEyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\)\. A CartridgeZ∈ℝL×p×d×2Z\\in\\mathbb\{R\}^\{L\\times p\\times d\\times 2\}is a compact, trainable KV cache that encodes a context ofn𝒞n\_\{\\mathcal\{C\}\}tokens as a fixed\-size prefix of lengthp≪n𝒞p\\ll n\_\{\\mathcal\{C\}\}that is injected into the model’s attention layersLL\(of sizedd\), yielding compression ratios of1010–100×100\\timeswhile integrating directly into existing inference servers\.

The Cartridge is trained via a context\-distillation objective using training data generated through*Self\-Study*: the corpus is chunked into subcorpora𝐜~\\tilde\{\\mathbf\{c\}\}, and the LLM generates synthetic question–answer pairs about each chunk using diverse seed prompts\(Eyubogluet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib8)\)\. The loss minimizes the KL divergence between a*teacher*\(a model with the subcorpus in context\) and a*student*\(the same model augmented with the Cartridge\)\. All model parameters remain frozen—only the key\-value vectors inZZreceive gradients\.

### 2\.1Training with Multiple Cartridges

AlthoughEyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\)discuss that independently trained Cartridges can be combined by concatenating their KV caches, this composition was never explicitly tested\. Figure[1](https://arxiv.org/html/2606.04557#S1.F1)shows that mixing caches trained*in isolation*—independently and without awareness of other Cartridges that may coexist at inference time—leads to model collapse\. Additionally, naive scaling to very large Cartridge sizes is not feasible since training over large document collections has high memory requirements\.111A 1K\-token Qwen3\-8B Cartridge takes∼551\\sim 551MiB during backprop \(500K tokens∼274\{\\sim\}274GiB\)\. Gradients and Adam moments increase the Cartridges’ memory footprint 4×\\times\.

To address these limitations, we introduce CAS, a training and inference framework that supports the use ofNNcartridges simultaneously \(Figure[2](https://arxiv.org/html/2606.04557#S1.F2)\)\.

#### Mixed\-visibility training\.

Mixed\-visibility training is key for learning KV representations that allow the frozen model to selectively attend to the relevant cartridge in the presence of distractors, directly optimizing for the multi\-cartridge inference setting\. To achieve this, we adapt the KL loss as follows: letZ∗∈\{Z1,…,ZNZ^\{\*\}\\in\\\{\{Z\_\{1\},\\ldots,Z\_\{N\}\}\} be the relevant cartridge for example𝐱\\mathbf\{x\},𝒵​pool\\mathcal\{Z\}\\text\{pool\}the cartridge pool,𝒵​dist=𝒵​pool∖Z∗\\mathcal\{Z\}\\text\{dist\}=\\mathcal\{Z\}\\text\{pool\}\\setminus\{Z^\{\*\}\}the distractor candidates, andk∼𝒰​\(kmin,kmax\)k\\sim\\mathcal\{U\}\(k\_\{\\text\{min\}\},k\_\{\\text\{max\}\}\), then the loss becomes:

Z~=\{Z∗w\.p\.​PisoZ∗∪S,S∼𝒰​\(𝒵dist,k\)w\.p\.​1−Piso\\tilde\{Z\}=\\resizebox\{325\.215pt\}\{\}\{$\\begin\{cases\}Z^\{\*\}&\\text\{w\.p\. \}P\_\{\\text\{iso\}\}\\\\ Z^\{\*\}\\cup S,\\;S\\sim\\mathcal\{U\}\(\\mathcal\{Z\}\_\{\\text\{dist\}\},k\)&\\text\{w\.p\. \}1\-P\_\{\\text\{iso\}\}\\end\{cases\}$\}\(1\)
ℒ\(Z\)=∑\(𝐱,𝐜~\)∈𝒟∑i=1\|𝐱\|DKL\(ℱ\(⋅∣𝐜~⊕𝐱<i\)∥ℱZ~\(⋅∣𝐱<i\)\)\\mathcal\{L\}\(Z\)=\\displaystyle\\sum\_\{\(\\mathbf\{x\},\\tilde\{\\mathbf\{c\}\}\)\\in\\mathcal\{D\}\}\\sum\_\{i=1\}^\{\|\\mathbf\{x\}\|\}D\_\{\\text\{KL\}\}\\\!\\left\(\\mathcal\{F\}\(\\cdot\\mid\\tilde\{\\mathbf\{c\}\}\\oplus\\mathbf\{x\}\_\{<i\}\)\\,\\\|\\,\\mathcal\{F\}\_\{\\tilde\{Z\}\}\(\\cdot\\mid\\mathbf\{x\}\_\{<i\}\)\\right\)

\(2\)
Training with packing requires keeping all active cartridges in the prompt, while each packed sample sees a different subset via an attention mask\.

#### Budget Manager

During training, we introduce a*budget manager*that maintains a fixed GPU pool ofB≤NB\\leq Ncartridges, keeping the remainingN−BN\-Bcartridges offloaded to CPU memory or disk \(Figure[2\(b\)](https://arxiv.org/html/2606.04557#S1.F2.sf2)\)\. At each step, the budget manager selects the active cartridges \(i\.e\., those residing on GPU\) and samples data from their training pool\.

#### Pool rotation\.

EveryRRoptimizer steps, we rotate the active cartridges in the GPU pool: a fractionϕ\\phiof the pool is evicted and replaced with new cartridges from persistent storage\. Evicted cartridges receive no gradients until they are moved back to GPU\. SettingR=1R=1andϕ=1\\phi=1rotates the entire pool at every step, maximizing diversity but incurring the highest data\-transfer overhead\. LargerRRor smallerϕ\\phiamortize this overhead by increasing the duration for which cartridges remain active, at the cost of slower turnover and reduced exposure to diverse cartridges during training\. The rotation policy preferentially swaps in cartridges that have received the fewest optimizer steps, ensuring uniform document coverage\.

To ensure the consistency of the RoPE embeddingsSuet al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib47)\), query token position IDs must be offset by the total number of active cartridge KV tokens, which occupy the prefix of the sequence\. This offset is recomputed at each step based on the actual number of tokens contributed by the active cartridges\.

### 2\.2Optimization Dynamics

#### Optimizer\.

We use AdamKingma and Ba \([2014](https://arxiv.org/html/2606.04557#bib.bib46)\)with fp32 precision, while we keep the weights in bfloat16, as this is crucial for numerical stability when scaling up the number of cartridges\.

Instead of a fixed learning rateEyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\), we adopt a slow linear decay schedule with per\-cartridge linear warmup to maintain a higher effective LR throughout training \(see Appendix[B](https://arxiv.org/html/2606.04557#A2)\)\.

#### Per\-cartridge LR warmup\.

When a cartridge is first swapped into the GPU pool, it has received zero optimizer steps and its gradients may be large relative to its current parameter values\. Applying the global LR immediately can destabilize training\. We therefore apply a per\-cartridge linear warmup: a cartridge that has receivedssoptimizer steps uses an effective LR ofηeff=ηt⋅min⁡\(1,sWc\)\\eta\_\{\\text\{eff\}\}=\\eta\_\{t\}\\cdot\\min\(1,\\frac\{s\}\{W\_\{c\}\}\), whereWcW\_\{c\}is the per\-cartridge warmup length\. To avoid creating one optimizer param group per cartridge \(which would prevent fused Adam kernels\), we bucket the warmup scale intoNb=4N\_\{b\}=4discrete levels, yielding at mostNbN\_\{b\}param groups per step\.

### 2\.3Improved Self\-Study Data Synthesis

The quality of synthetic training data is the primary driver of cartridge performance\. A detailed analysis on FinQA documents \(Appendix[J](https://arxiv.org/html/2606.04557#A10)\) reveals three key failures of the original procedure: \(*i*\)*uneven fact coverage*—since questions are generated without global awareness of the document, the synthesizer focuses on prose commentary rather than details such as figures in tables; \(*ii*\)*unbalanced document sampling*—facts are sampled uniformly across chunks, causing facts in longer, denser sections to be under\-represented; and \(*iii*\)*low throughput*—one question per API call makes synthesis slow and expensive\. To mitigate these limitations, we propose proportional\-to\-length sampling and batched multi\-question generation \(Figure[2\(a\)](https://arxiv.org/html/2606.04557#S1.F2.sf1)\)\.

#### Proportional\-to\-length sampling\.

We replace uniform random chunk sampling with a*proportional\-to\-length*strategy ensuring balanced document coverage\. Each documentdid\_\{i\}in the collection is assigned a sampling weightwi=\|di\|minj⁡\|dj\|w\_\{i\}=\\frac\{\|d\_\{i\}\|\}\{\\min\_\{j\}\|d\_\{j\}\|\}, so that longer documents—which are assumed to contain more facts—are sampled proportionally more often during synthesis\.

#### Multi\-question generation\.

We prompt the question\-generation model to producennquestions simultaneously \(instead of doingnnLLM calls\); we usen=20n\{=\}20and sample multiple times with temperature 0\.6 \(Figure[2\(a\)](https://arxiv.org/html/2606.04557#S1.F2.sf1)\)\. The prompt instructs the model to vary the question style \(factual recall, comparison, reasoning, detail\-oriented\) and to cover different facts, details, or aspects of the provided context\. Responses are returned as an array, parsed into individual questions and answered independently by the student model\.

#### Decoupled question generation\.

We decouple the question\-generation modelMQM\_\{Q\}from the answer\-generationMAM\_\{A\}\.MQM\_\{Q\}is a larger, more capable model\(Agarwalet al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib23), GPT\-OSS 120B\)that generates high\-quality, diverse questions given the document context, whereasMAM\_\{A\}remains the target model whose KV cache we want to learn; it produces answers conditioned on the document—along with their token\-level log\-probabilities—as distillation targets for cartridge training \(details in Appendix[C](https://arxiv.org/html/2606.04557#A3)\)\.

### 2\.4Cartridge Initialization

KV cache initialization quality matters for final Cartridge performance\.Eyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\)show that initializing the cartridge using the KV cache from the firstpptokens of an arbitrary text outperforms random initialization, as it already projects the KVs within the model’s latent space\.

#### Cart\-specific initialization\.

We propose a simple yet effective change to this procedure: when training over a collection ofNNdocuments, each cartridgeZiZ\_\{i\}is initialized from its own truncated documentdid\_\{i\}rather than from an arbitrary text\. This ensures that the initial KV cache already encodes document\-specific content, providing a warm start that significantly reduces the initial loss \(over∼\\sim50% lower in our experiments\), reduces the number of training steps required to reach convergence and leads to∼\\sim10% lower final loss\.

We experimented with pooling\-based compression strategies \(mean/max pooling, top\-kknorm selection, and sliding\-window averaging\) as alternatives for capturing more information from the original document, but none outperformed the cart\-specific truncation strategy\.

DatasetDocsQuestionsQs/DocAvg\. Tok\.Total Tok\.Task TypeDomainLongHealth2040020\.011,700236KMultiple\-choice \(5\-way\)Clinical patient recordsQASPER4071,4513\.54,751665KExtract\./free\-form/yes\-noFull research papersQuALITY1152,08618\.15,7131\.9MMultiple\-choice \(4\-way\)Fiction & non\-fiction narrativesT2\-RB/FinQA3801,1473\.01,026392KMath\. CalculationCorporate earnings reports w/ tablesTechQA4966101\.81,509748KExtractiveIBM IT support technotes

Table 1:Dataset statistics\.Docs: unique documents \(one cartridge per document\)\.Qs/Doc: average questions per document\.Avg\. Tok\.: mean document length in tokens \(Qwen3\-8B tokenizer\)\.

### 2\.5Inference

At inference time, the model has access to a collection ofNNtrained cartridges\{Z1,…,ZN\}\\\{Z\_\{1\},\\ldots,Z\_\{N\}\\\}, each encoding a distinct documentdid\_\{i\}\. When combining multiple cartridges we prepend their sequentially concatenated KV caches to the formatted prompt and question \(Figure[2\(c\)](https://arxiv.org/html/2606.04557#S1.F2.sf3)\), without stripping any special tokens from the processed inputs\.

Because loading allNNcartridges simultaneously is both inefficient and infeasible for large collections \(prefix length grows as𝒪​\(N⋅p\)\\mathcal\{O\}\(N\\cdot p\)\), we select thekkmost relevant cartridges for a given queryqqbefore the forward pass\. We evaluate two selection strategies: \(*i*\)*Oracle*, where the ground\-truth cartridge is always loaded \(an upper bound, not applicable in practice\), and \(*ii*\)*Retrieved*, where the top\-kkcartridges are selected by a dense retriever\.

## 3Experimental Setup

#### Models\.

We use Qwen3\-8B\(Yanget al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib22)\)as the main model in our experiments \(see Appendix[A](https://arxiv.org/html/2606.04557#A1)for training and inference details\)\. Our RAG baseline uses dense retrieval via Amazon Bedrock Knowledge Bases \(details in Appendix[F](https://arxiv.org/html/2606.04557#A6)\)\.

#### Datasets\.

We use these 5 datasets \(Table[1](https://arxiv.org/html/2606.04557#S2.T1)\):222We filter out unanswerable questions and outlier documents that are significantly longer than others in the collection\.

- •LongHealthAdamset al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib14)\): A clinical QA benchmark consisting of detailed fictional patient records with multiple\-choice questions that test information extraction, negation understanding, and temporal sorting over long medical notes\.
- •QASPERDasigiet al\.\([2021](https://arxiv.org/html/2606.04557#bib.bib15)\): An info\-seeking QA dataset over full NLP research papers, where questions are written by readers who only saw the title and abstract\. Answers include extractive spans, free\-form text, and yes/no responses\.
- •QuALITYPanget al\.\([2022](https://arxiv.org/html/2606.04557#bib.bib16)\): A multiple\-choice reading comprehension benchmark over long\-form fiction and non\-fiction narratives, where questions require careful reading and reasoning rather than surface\-level pattern matching\.
- •T2\-RAGBench/FinQAStrichet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib13)\); Chenet al\.\([2021](https://arxiv.org/html/2606.04557#bib.bib12)\): A decontextualized variant of FinQA designed for RAG evaluation, where questions about corporate earnings reports have been rewritten to be context\-independent\. The task requires numerical reasoning by constructing mathematical formulas over tables and text extracted from financial filings\.
- •TechQACastelliet al\.\([2020](https://arxiv.org/html/2606.04557#bib.bib17)\): A technical support QA dataset drawn from IBM’s Technote corpus, where questions require extracting precise solutions from IT documentation covering enterprise software and infrastructure issues\.

MethodComp\.LongHealth \(Acc\.\)QuALITY\(Acc\.\)QASPER\(F1\)FinQA\(EM\)TechQA\(LLaaJ\)No Context–37\.5±\\pm1\.143\.6±\\pm0\.419\.2±\\pm0\.62\.9±\\pm0\.021\.1±\\pm1\.1Oracle Context1×1\\times87\.4±\\pm0\.882\.5±\\pm0\.356\.7±\\pm0\.466\.8±\\pm2\.774\.7±\\pm0\.9CAS Training \(Ours\)Single\-Cartridge / Doc\.2×2\\times81\.1±\\pm1\.178\.6±\\pm0\.954\.9∗±\\pm0\.362\.7±\\pm0\.275\.8±\\pm0\.75×5\\times79\.9±\\pm0\.878\.5±\\pm0\.954\.9±\\pm0\.657\.9±\\pm0\.475\.6±\\pm0\.710×10\\times80\.1±\\pm0\.877\.6±\\pm0\.554\.1±\\pm0\.350\.7±\\pm0\.875\.2±\\pm2\.020×20\\times78\.9±\\pm1\.077\.6±\\pm0\.653\.9±\\pm0\.645\.0±\\pm0\.975\.0±\\pm0\.450×50\\times76\.9±\\pm2\.276\.7±\\pm0\.553\.1±\\pm0\.430\.2±\\pm1\.874\.6±\\pm1\.5100×100\\times77\.3±\\pm1\.476\.4±\\pm0\.853\.1±\\pm0\.323\.0±\\pm0\.374\.7±\\pm0\.9

Table 2:No Context, Oracle Context, and Oracle Cartridge at selected compression ratios on Qwen3\-8B\.±\\pmdenotes std\. dev\. over 3 runs\.Boldmarks the best cartridge result per dataset\.∗Trained with 3×\\timesdue to memory constraints\.
#### Retrieval\.

For our baseline RAG system, we use a chunk\-based indexing of the documents, then we retrieve the relevant chunks based on the question, without reformulation \(details in Appendix[F](https://arxiv.org/html/2606.04557#A6)\)\. In the cartridge case, we use the RAG results as a proxy to retrieve relevant cartridges \(see Figure[2\(c\)](https://arxiv.org/html/2606.04557#S1.F2.sf3)\)\. In particular, we map each retrieved chunk to a document id, which in turn is associated with a cartridge\. The final list contains the unique cartridge ids in the order of their retrieval\. With 1,024\-token chunks, LongHealth and QuALITY average∼2\{\\sim\}2chunks per unique cartridge, while for TechQA and FinQA the ratio is closer to 1:1 \(see Table[9](https://arxiv.org/html/2606.04557#A6.T9)\)\.

#### Evaluation

Each dataset uses a task\-specific system prompt during generation to elicit answers in the expected format\. For FinQA, the model is instructed to produce structured mathematical formulas \(e\.g\.,divide\(subtract\(x, y\), y\)\)\. For LongHealth and QuALITY, the model selects from multiple\-choice options\. For QASPER, the model provides brief extractive or yes/no answers\. For TechQA, the model needs to produce concise factual answers\. The full model prompts that we use are provided in Appendix[H](https://arxiv.org/html/2606.04557#A8)\.

## 4Experimental Results

SettingOracle \(1 doc\)Full \(20 docs\)Oracle Context87\.4±\\pm0\.8–†/ 65\.5Train in Isolation73\.6±\\pm0\.426\.0±\\pm3\.1Train Jointly \(25% ex\., 10 act\.\)78\.9±\\pm1\.056\.4±\\pm0\.6Train Jointly \(25% ex\., 20 act\.\)79\.0±\\pm1\.477\.8±\\pm0\.6

Table 3:Effects of the training regime on inference\.*Oracle*– only the gold cartridge;*Full*– all notes simultaneously\.±\\pmdenotes std\. dev\. over 3 runs\.†All notes exceeds the context; 65\.5 uses 10 patients groups\.#### Training Regime\.

We compare cartridge performance for LongHealth in two regimes: \(*i*\) all cartridges are trained in isolation \(Piso=1\.0P\_\{\\text\{iso\}\}\{=\}1\.0\), and \(*ii*\) they are mixed in 1 out of every 4 examples \(Piso=0\.75P\_\{\\text\{iso\}\}\{=\}0\.75\)\. In both regimes, we train with the same hyper\-parameters—compression of20×20\\times\(∼585\{\\sim\}585tokens/cart\.\), 80 epochs, and addingk∼𝒰​\(1,M\)k\\sim\\mathcal\{U\}\(1,M\)additional cartridges as distractors\.

First, we focus on the oracle setting, where at inference time each question is given only the relevant cartridge in the KV prefix\. In Table[3](https://arxiv.org/html/2606.04557#S4.T3)\(*Oracle*\), the two joint training regimes achieve comparable performance—78\.978\.9\(M=10M=10active\) and79\.079\.0\(M=20M=20\)—while isolated training remains at73\.673\.6\. This suggests that single\-cartridge quality benefits from exposure to distractors during training\.

The gap widens drastically when we decode with all patient cartridges \(*Full*\): joint training with 10 active cartridges degrades to56\.456\.4, while isolated training collapses to26\.026\.0\. This shows that mixing cartridges trained in isolation leads to catastrophic interference, since the model never learned to distinguish between the distributions encoded in individual cartridges\. Crucially, joint training with all 20 cartridges active nearly eliminates this degradation: the*Full \(20 docs\)*score reaches77\.877\.8, only1\.21\.2below its Oracle Cartridge result\.

#### Positional Invariance\.

Figure[3](https://arxiv.org/html/2606.04557#S4.F3)quantifies how the performance changes under different permutations of the cartridges\. We see that shuffling the cartridge order has no effect \(77\.877\.8both for*Ordered*and*Shuffled*\), demonstrating position\-invariant attention over the KV prefix\. When a random subset ofkkcartridges is loaded, accuracy on questions whose cartridge*is*present remains high \(7878–8787\), while accuracy on questions whose cartridge is*absent*drops to2727–3131\. This confirms that the model attends selectively to the correct cartridge rather than relying on a generic shared memory\. Even though we back\-propagate through all active cartridges simultaneously, there is a clear knowledge separation: each cartridge encodes document\-specific information that the model can retrieve independently of the other cartridges in the prefix\.

![Refer to caption](https://arxiv.org/html/2606.04557v1/images/selectivity_matrix_act20.png)Figure 3:Accuracy of*Train Jointly \(25% ex\., 20 act\.\)*on LongHealth varying the number of present cartridges\.
#### Scaling to Hundreds of Cartridges\.

Next, we scale up training to four additional datasets, each containing several hundred unique documents \(Table[1](https://arxiv.org/html/2606.04557#S2.T1)\)\. In this setting, we simulate realistic document collections with up to a million tokens\.

Table[2](https://arxiv.org/html/2606.04557#S3.T2)compares the model performance when consuming raw contexts vs\. their compressed variants in individual cartridges\. The*No Context*row is the model performance without context, only using the question and the system prompt\. We can see that all datasets perform poorly compared to the*Oracle Context*, showing that the model is not able to solve the tasks parametrically\. In the second part of the table, we show the best results at a given target compression \(2×2\\timesto100×100\\times\)333Each document is compressed to its length divided by the target compression\.for each dataset\.

![Refer to caption](https://arxiv.org/html/2606.04557v1/x2.png)Figure 4:Accuracy vs\. additional prompt tokens trade\-off between text and Cartridge RAG\. The red star denotes the text oracle score;tt: compression ratio;kk: number of retrieved chunks\.We see the following patterns across datasets: \(*i*\) the achievable compression depends strongly on document complexity: factually dense datasets and longer documents \(LongHealth\) do not tolerate high compression rates, while extraction tasks that do not require reasoning \(TechQA\) have no loss in performance even at100×100\\timescompression; \(*ii*\) the parametric knowledge of the model is stabilizing the compression – QuALITY and QASPER contain books/papers that are included in many pre\-training corporaGaoet al\.\([2020](https://arxiv.org/html/2606.04557#bib.bib20)\); Weberet al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib21)\)and have only a low drop at higher compressions, \(*iii*\) dense structures in the text, such as tables, are hard to compress, i\.e\., FinQA drops 4\-9 points at2×2\\times\-5×5\\timescompression, similar to LongHealth\.

Token BudgetLongHealthQuALITYFinQACompr\.Acc\.Compr\.Acc\.Compr\.EM4,09657×\\times67\.2160×\\times62\.196×\\times6\.920,00012×\\times70\.533×\\times64\.220×\\times14\.128,6728×\\times72\.623×\\times65\.214×\\times13\.0Multi\-cart\.10×\\times80\.120×\\times77\.620×\\times45\.0

Table 4:Performance of a monolith cartridge trained on all documents under varying budgets\.
#### Cartridges Granularity\.

We ablate whether the gains come from*compression capacity*\(the cartridge has≥p\\geq pKV slots\) or from*per\-document specialization*\(each document has its own cartridge\)\. We disentangle the two by training a single monolithic cartridge on the concatenation of all documents, with cartridge sizes of 4K, 20K, 28\.7K tokens—the latter being roughly the full available KV budget given a3232K context window \(no YaRNPenget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib71)\)\) minus prompt and generation overhead\.

Despite allocating up to 28K tokens, the monolith baseline \(Table[4](https://arxiv.org/html/2606.04557#S4.T4)\) achieves only72\.672\.6on LongHealth,65\.265\.2on QuALITY, and14\.114\.1on FinQA\. In contrast, per\-document cartridges \(*Multi\-cart\.*\) reach80\.180\.1on LongHealth at10×10\\timescompression \(∼1\.2\{\\sim\}1\.2K tok\./doc,23\.623\{\.\}6K total\),78\.678\.6on QuALITY at2×2\\times\(∼2,9\{\\sim\}2\{,\}9K tok\./doc\), and62\.762\.7on FinQA at2×2\\times\(∼524\{\\sim\}524tok\./doc\)\. This large gap confirms that per\-document specialization is critical\. The FinQA results are particularly striking: 380 financial documents with dense numerical content cannot be meaningfully compressed into a single shared representation, as gradient updates from different documents destructively interfere\. Nonetheless, dedicating a separate cartridge to each doc resolves this interference entirely in the oracle setting\.

#### Cartridge RAG\.

The*Oracle*results assume the gold cartridge is always loaded, but in practice we often do not know which document or context is relevant\. Moreover, performing inference with hundreds or thousands of cartridges in the prompt is inefficient and even infeasible\. Here, we propose a hybrid approach combining standard dense retrieval with cartridge inference: given a query, we retrieve the top\-kkdocument chunks, resolve the corresponding cartridge identifiers, and load only those cartridges into the KV prefix instead of the raw chunk text\. While*Text RAG*appendskkchunks of∼1,024\{\\sim\}1\{,\}024tokens each \(consumingk×1​Kk\{\\times\}1\\text\{K\}prompt tokens; see Appendix[F](https://arxiv.org/html/2606.04557#A6)\),*Cartridge RAG*loads the corresponding document cartridges att×t\{\\times\}compression, e\.g\., att=20t\{=\}20, a1212K\-token document occupies only∼600\{\\sim\}600KV tokens, representing the*entire*document rather than a single chunk fragment\.

Figure[4](https://arxiv.org/html/2606.04557#S4.F4)reports accuracy as a function of the additional prompt tokens consumed at inference\.444We exclude QASPER from the RAG comparison; the data is not decontextualized \(“What datasets do they use?”\)\.On LongHealth and QuALITY, where retrieval recall is high \(≥95%\{\\geq\}95\\%atk=10k\{=\}10; Appendix[G](https://arxiv.org/html/2606.04557#A7)\),*Cartridge RAG*clearly dominates*Text RAG*\. On LongHealth,*Cartridge RAG*att=20t\{=\}20,k=10k\{=\}10matches the best*Text RAG*\(75\.475\.4\) using only2,6732\{,\}673tokens—3\.7×3\.7\\timesfewer than Text RAG’s9,8609\{,\}860—and att=100t\{=\}100,k=10k\{=\}10still delivers76\.576\.5at just566566tokens\. On QuALITY, Cartridge RAG witht=5t\{=\}5,k=5k\{=\}5reaches75\.375\.3, surpassing Text RAG \(73\.973\.9\) by1\.41\.4at roughly half the token budget \(2,6352\{,\}635vs\.4,7544\{,\}754\)\. On FinQA, we observe a clear trend where loading more cartridges*hurts*: att=2t\{=\}2, accuracy peaks atk=5k\{=\}5\(52\.852\.8\) and drops to50\.750\.7atk=10k\{=\}10, consistently across all compression ratios\. We hypothesize that the model becomes overwhelmed by the dense numerical content encoded in multiple financial\-document cartridges, and that increasing the number of active cartridges during training \(currently capped at 10\) may alleviate this\. On TechQA,*Text RAG*slightly outperforms*Cartridge RAG*\(73\.273\.2vs\.71\.071\.0\), likely because answers require extracting precise spans from the gold text—a setting where verbatim token access in raw chunks is more effective than compressed KV representations from multiple documents that bring noise\. Still, in the TechQA case*Cartridge RAG*remains more effective until retrieving more than 5 chunks\.

## 5Discussion

#### Cost Analysis\.

The main cost of using cartridges lies in the offline training process which includes self\-study data generation: sampling 100\-200 questions/doc\. and doing one inference with the target model to generate the response555We are modeling the KL divergence which allows for training on partial sequences, unlike in cross\-entropy loss\.\. Depending on the size, complexity, and information density of documents, the training epochs can vary but often around 10 to 20 epochs are enough to reach 95% of the best cartridge performance \(Figure[1](https://arxiv.org/html/2606.04557#S1.F1)\)\.

Inference costs are limited to loading the cartridges; there are no additional inference costs since we do not need any prefills\. At inference, since prefill cost scales as the attention cost \(𝒪​\(n2\)\\mathcal\{O\}\(n^\{2\}\)\), a10×10\\timestoken reduction yields∼100×\{\\sim\}100\\timesfewer prefill FLOPs: Cartridge atr=10r\{=\}10uses∼1,200\{\\sim\}1\{,\}200tokens vs\.∼12,000\{\\sim\}12\{,\}000for the text Oracle and∼9,860\{\\sim\}9\{,\}860for Text RAG \(k=10k\{=\}10\) on LongHealth\.

Storage is another factor to consider\. The KV cache footprint in BF16 per 1K tokens scales with model size:∼110\{\\sim\}110MiB for Qwen3\-0\.6B,∼141\{\\sim\}141MiB for 4B/8B, and∼250\{\\sim\}250MiB for 32B\.

#### Document grouping\.

Our proposed CAS framework naturally extends to*grouped cartridges*encoding multiple related documents \(e\.g\., quarterly reports from the same company, chapters of the same book\)\. Grouping exploits shared vocabulary and background knowledge, allowing the token budget to focus on unique information that can help achieve even higher compression\. It also addresses the retrieval recall ceiling on homogeneous corpora: by clustering co\-relevant documents, the retriever operates over a smaller, more discriminative set of groups, and a single retrieved group cartridge provides richer context than any single\-document cartridge\. We leave the systematic study of grouping strategies to future work\.

## 6Related Work

#### Soft Prompting\.

These methods are variants of prefix\-tuning methods\(Lesteret al\.,[2021](https://arxiv.org/html/2606.04557#bib.bib67)\)that compress a given text into continuous representationsLiet al\.\([2025a](https://arxiv.org/html/2606.04557#bib.bib66)\)\. Similar to KV cache compression, soft compression encodes long contexts into compact soft representations that can be prepended to queries, allowing for storing and caching\. AutoCompressors\(Chevalieret al\.,[2023](https://arxiv.org/html/2606.04557#bib.bib41)\)recursively generate summary vectors; Gisting\(Bulatovet al\.,[2022](https://arxiv.org/html/2606.04557#bib.bib63); Muet al\.,[2023](https://arxiv.org/html/2606.04557#bib.bib64)\)condenses prompts into virtual gist tokens; and other methods encode contexts into compact memory slots via LoRA\-adapted or KV\-based encoders\(Geet al\.,[2024b](https://arxiv.org/html/2606.04557#bib.bib68); Liet al\.,[2025b](https://arxiv.org/html/2606.04557#bib.bib69)\)\. xRAG\(Chenget al\.,[2024](https://arxiv.org/html/2606.04557#bib.bib32)\)and PISCO\(Louiset al\.,[2025](https://arxiv.org/html/2606.04557#bib.bib33)\)take a modular approach, projecting retrieval embeddings or distilled representations into the LLM’s input space while keeping the decoder frozen\. These methods differ primarily in their training objectives \(KL divergence vs\. sequence\-level distillation\) and in which components remain trainable\.

#### KV Cache Compression\.

A natural way to reduce the context is to apply operations to its KV cache\. This can happen either after or during the prefill phase by evicting non\-impactful key\-value pairsZhanget al\.\([2023](https://arxiv.org/html/2606.04557#bib.bib34)\); Orenet al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib36)\); Geet al\.\([2024a](https://arxiv.org/html/2606.04557#bib.bib59)\); Tanget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib60)\); Chari and Durme \([2025](https://arxiv.org/html/2606.04557#bib.bib61)\); Łańcuckiet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib62)\)or merging themWanget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib55)\); Zhanget al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib56)\); Liuet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib57)\); Wanet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib58)\)\. SnapKVLiet al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib53)\)and PyramidKVCaiet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib54)\)evict low\-importance KV entries based on attention scores observed during prefill, with PyramidKV additionally allocating non\-uniform per\-layer budgets\. KVzipKimet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib43)\)selects tokens to retain via a learned context\-reconstruction objective\. Attention MatchingZweigeret al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib44)\)optimizes a compacted cache to reproduce the original attention distribution instead of reconstructing individual KV vectors\. RetrievalAttentionLiuet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib35)\)takes an orthogonal approach, offloading the full KV cache to the CPU and using approximate nearest\-neighbor search to fetch the relevant entries\.

We view many of these KV cache techniques as complementary to cartridges: compacted caches can serve as a more effective initialization point, which is then fine\-tuned to the collection distribution; additionally, compaction can be applied to the trained cartridges because their representation is query\-independent and may admit further compression for specific queries\.

## 7Conclusions

We present an end\-to\-end framework for training and serving multiple cartridges at a granularity as small as that of individual documents\. Our framework addresses a fundamental flaw in the original monolithic Cartridges design, which prevents mixing independently trained cartridges, leading to collapse in performance at near chance levels\. Experiments across five long\-context datasets demonstrate that our approach scales to hundreds of cartridges over hundreds of thousands to millions of tokens, outperforming monolithic designs by 10–30 points at comparable computational budgets\. We further propose improved initialization strategies and more efficient self\-study generation, achieving superior accuracy and efficiency\. Compression tolerance varies with document complexity \(100×100\\timesfor TechQA,≤2×\{\\leq\}2\\timesfor FinQA\), reaching within 5% of uncompressed baselines\. Finally, we show that cartridges can be combined with retrieval, matching RAG performance while using 3–4×\\timesfewer tokens, though for information\-dense tasks such as FinQA, text RAG maintains an edge\.

## Acknowledgments

We want to thank Rexhina Blloshmi, Felix Hieber, Bill Byrne, Hagen Fürstenau and Karan Gill for the time they took to support, discuss and improve this work\.

## Limitations

#### Cartridge KV position\.

All of our experiments prepend the trained KV cache to the prompt and query tokens, which is the natural setting for single\-turn QA: the cartridge occupies the earliest positions and all subsequent tokens attend to it via standard causal masking\. In multi\-turn scenarios, however, a cartridge may need to be loaded mid\-conversation—for example, when a user references a new document after several turns of dialogue\. Prepending the cartridge at that point would invalidate the previously computed KV entries, requiring a full recomputation of the conversation history\.

#### Cartridge\-specific retrieval\.

Our Cartridge RAG pipeline reuses a standard text\-based dense retriever to select which cartridges to activate\. This is a pragmatic choice, but it means retrieval quality is bounded by how well the query matches the*raw document text*rather than the compressed KV representation\. A dedicated cartridge retriever—one that learns to match queries directly against cartridge embeddings—could improve recall, particularly on corpora where text\-based retrieval is unreliable\. Developing such a retriever is a natural next step, but requires either a learned projection from cartridge KV space to a query\-compatible embedding space, or a contrastive training objective that aligns query representations with cartridge representations\.

#### Language coverage\.

All five benchmarks used in this work are in English, and the base model \(Qwen3\-8B\) is predominantly trained on English text\. It is unclear whether the self\-study synthesis procedure, which relies on the model’s own generative capabilities to produce training data, transfers equally well to lower\-resource languages where the model’s generation quality is weaker\. We do not evaluate cartridge performance on non\-English corpora and cannot make claims about multilingual generalization\.

#### Model scale\.

All experiments use a single model size\. Larger models may achieve higher compression ratios at the same quality level, since their greater parametric capacity can compensate for more aggressive KV compression\. Conversely, smaller models may require lower compression ratios to retain acceptable performance\. Nonetheless, previous workEyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\); Zweigeret al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib44)\)showed that cartridge performance scales across model sizes and model families\.

## Potential Risks

Cartridges enable efficient, persistent encoding of arbitrary document collections into reusable KV caches\. A malicious actor could exploit this to encode and serve disinformation, harmful instructions, or private data at scale\. Additionally, documents encoded into cartridges may retain sensitive information in their KV representations in ways that are difficult to audit or redact—unlike raw text retrieval, it is unclear to what extent personal or confidential content can be extracted from a trained cartridge, raising data governance concerns in enterprise or medical deployments\. We encourage practitioners to apply the same content\-filtering, bias mitigation, and access\-control measures to cartridge collections as they would to the underlying documents\.

## References

- LongHealth: a question answering benchmark with long clinical documents\.arXiv preprint arXiv:2401\.14490\.Cited by:[1st item](https://arxiv.org/html/2606.04557#A11.I1.i1.p1.1),[Figure 1](https://arxiv.org/html/2606.04557#S1.F1),[1st item](https://arxiv.org/html/2606.04557#S3.I1.i1.p1.1)\.
- S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[Appendix I](https://arxiv.org/html/2606.04557#A9.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.04557#S1.p1.1),[§2\.3](https://arxiv.org/html/2606.04557#S2.SS3.SSS0.Px3.p1.4)\.
- Anthropic \(2025\)System card: Claude opus 4 & Claude sonnet 4\.Technical reportAnthropic\.External Links:[Link](https://anthropic.com/)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- A\. Bulatov, Y\. Kuratov, and M\. Burtsev \(2022\)Recurrent memory transformer\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/47e288629a6996a17ce50b90a056a0e1-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: Dynamic KV cache compression based on pyramidal information funneling\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=ayi7qezU87)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- V\. Castelli, R\. Chakravarti, S\. Dana, A\. Ferritto, R\. Florian, M\. Franz, D\. Garg, D\. Khandelwal, S\. McCarley, M\. McCawley, M\. Nasr, L\. Pan, C\. Pendus, J\. Pitrelli, S\. Pujar, S\. Roukos, A\. Sakrajda, A\. Sil, R\. Uceda\-Sosa, T\. Ward, and R\. Zhang \(2020\)The TechQA dataset\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 1269–1278\.External Links:[Link](https://aclanthology.org/2020.acl-main.117/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.117)Cited by:[5th item](https://arxiv.org/html/2606.04557#A11.I1.i5.p1.1),[5th item](https://arxiv.org/html/2606.04557#S3.I1.i5.p1.1)\.
- V\. Chari and B\. V\. Durme \(2025\)Compactor: Calibrated query\-agnostic KV cache compression with approximate leverage scores\.External Links:[Link](https://arxiv.org/abs/2507.08143),2507\.08143Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Chen, W\. Chen, C\. Smiley, S\. Shah, I\. Borova, D\. Langdon, R\. Moussa, M\. Beane, T\. Huang, B\. Routledge, and W\. Y\. Wang \(2021\)FinQA: a dataset of numerical reasoning over financial data\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Online and Punta Cana, Dominican Republic,pp\. 3697–3711\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.300/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.300)Cited by:[4th item](https://arxiv.org/html/2606.04557#A11.I1.i4.p1.1),[4th item](https://arxiv.org/html/2606.04557#S3.I1.i4.p1.1)\.
- X\. Cheng, X\. Wang, X\. Zhang, T\. Ge, S\. Chen, F\. Wei, H\. Zhang, and D\. Zhao \(2024\)XRAG: extreme context compression for retrieval\-augmented generation with one token\.Advances in Neural Information Processing Systems37,pp\. 109487–109516\.Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Chevalier, A\. Wettig, A\. Ajith, and D\. Chen \(2023\)Adapting language models to compress contexts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 3829–3846\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.232),[Link](https://aclanthology.org/2023.emnlp-main.232/)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. Gardner \(2021\)A dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 4599–4610\.Cited by:[2nd item](https://arxiv.org/html/2606.04557#A11.I1.i2.p1.1),[2nd item](https://arxiv.org/html/2606.04557#S3.I1.i2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\.arXiv preprint arXiv:2501\.12948\.External Links:[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- A\. Devoto, M\. Jeblick, and S\. Jégou \(2025\)Expected attention: kv cache compression by estimating attention from future queries distribution\.arXiv preprint arXiv:2510\.00636\.External Links:[Link](https://arxiv.org/abs/2510.00636)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p4.2)\.
- S\. Eyuboglu, R\. S\. Ehrlich, S\. Arora, N\. Guha, D\. Zinsley, E\. R\. Liu, A\. Rudra, J\. Y\. Zou, A\. Mirhoseini, and C\. Re \(2025\)Cartridges: lightweight and general\-purpose long context representations via self\-study\.InES\-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models,External Links:[Link](https://openreview.net/forum?id=DuVSIWY5vC)Cited by:[1st item](https://arxiv.org/html/2606.04557#A3.I1.i1.p1.1),[Appendix H](https://arxiv.org/html/2606.04557#A8.p2.1),[§1](https://arxiv.org/html/2606.04557#S1.p4.2),[§2](https://arxiv.org/html/2606.04557#S2.SS0.SSS0.Px1.p1.7),[§2](https://arxiv.org/html/2606.04557#S2.SS0.SSS0.Px1.p2.2),[§2\.1](https://arxiv.org/html/2606.04557#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2606.04557#S2.SS2.SSS0.Px1.p2.1),[§2\.4](https://arxiv.org/html/2606.04557#S2.SS4.p1.1),[Model scale\.](https://arxiv.org/html/2606.04557#Sx2.SS0.SSS0.Px4.p1.1)\.
- L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima,et al\.\(2020\)The pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§4](https://arxiv.org/html/2606.04557#S4.SS0.SSS0.Px3.p3.3)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[Appendix H](https://arxiv.org/html/2606.04557#A8.p2.1)\.
- S\. Ge, Y\. Zhang, L\. Liu, M\. Zhang, J\. Han, and J\. Gao \(2024a\)Model tells you what to discard: adaptive KV cache compression for LLMs\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uNrFpDPMyo)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- T\. Ge, H\. Jing, L\. Wang, X\. Wang, S\. Chen, and F\. Wei \(2024b\)In\-context autoencoder for context compression in a large language model\.InThe Twelfth International Conference on Learning Representations,ICLR ’24,Vienna, Austria\.External Links:[Link](https://openreview.net/forum?id=uREj4ZuGJE)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)Retrieval augmented language model pre\-training\.InInternational conference on machine learning,pp\. 3929–3938\.Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p2.1)\.
- H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu \(2023\)LLMLingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 13358–13376\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.825),[Link](https://aclanthology.org/2023.emnlp-main.825)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient kv cache compression recipe for near\-lossless generative inference of llm\.arXiv preprint arXiv:2403\.05527\.Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1)\.
- J\. Kim, J\. Kim, S\. Kwon, J\. W\. Lee, S\. Yun, and H\. O\. Song \(2026\)KVzip: query\-agnostic KV cache compression with context reconstruction\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=JFygzwx8SJ)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p4.2),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§2\.2](https://arxiv.org/html/2606.04557#S2.SS2.SSS0.Px1.p1.1)\.
- W\. Łajewska, M\. Hardalov, L\. Aina, N\. Anna John, H\. Su, and L\. Marquez \(2025\)Understanding and improving information preservation in prompt compression for LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 17520–17541\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.949),ISBN 979\-8\-89176\-335\-7,[Link](https://aclanthology.org/2025.findings-emnlp.949/)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1)\.
- A\. Łańcucki, K\. Staniszewski, P\. Nawrot, and E\. Ponti \(2025\)Inference\-time hyper\-scaling with KV cache compression\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8ZiElzQxf1)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- B\. Lester, R\. Al\-Rfou, and N\. Constant \(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Online and Punta Cana, Dominican Republic,pp\. 3045–3059\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.243),[Link](https://aclanthology.org/2021.emnlp-main.243)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p2.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/28ab418242603e0f7323e54185d19bde-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Li, Y\. Liu, Y\. Su, and N\. Collier \(2025a\)Prompt compression for large language models: a survey\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 7182–7195\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.368),ISBN 979\-8\-89176\-189\-6,[Link](https://aclanthology.org/2025.naacl-long.368/)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Li, Y\. Su, and N\. Collier \(2025b\)500xCompressor: generalized prompt compression for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 25081–25091\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1219)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang,et al\.\(2026\)Retrievalattention: accelerating long\-context llm inference via vector retrieval\.Advances in Neural Information Processing Systems38,pp\. 54358–54385\.Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- X\. Liu, X\. Wang, P\. Liu, and G\. Tang \(2025\)ZSMerge: zero\-shot KV cache compression for memory\-efficient long\-context LLMs\.External Links:[Link](https://arxiv.org/abs/2503.10714),2503\.10714Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- M\. Louis, H\. Déjean, and S\. Clinchant \(2025\)PISCO: pretty simple compression for retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 15506–15521\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.800),ISBN 979\-8\-89176\-256\-5,[Link](https://aclanthology.org/2025.findings-acl.800/)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Mu, X\. Li, and N\. Goodman \(2023\)Learning to compress prompts with gist tokens\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/3d77c6dcc7f143aa2154e7f4d5e22d68-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2024\)OpenAI o1 System Card\.arXiv preprint arXiv:2412\.16720\.External Links:[Link](https://arxiv.org/abs/2412.16720)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- OpenAI \(2025\)OpenAI o3 and o4\-mini System Card\.Note:[https://cdn\.openai\.com/pdf/2221c875\-02dc\-4789\-800b\-e7758f3722c1/o3\-and\-o4\-mini\-system\-card\.pdf](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- M\. Oren, M\. Hassid, N\. Yarden, Y\. Adi, and R\. Schwartz \(2024\)Transformers are multi\-state RNNs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 18724–18741\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1043),[Link](https://aclanthology.org/2024.emnlp-main.1043/)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Pan, Q\. Wu, H\. Jiang, M\. Xia, X\. Luo, J\. Zhang, Q\. Lin, V\. Rühle, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and D\. Zhang \(2024\)LLMLingua\-2: data distillation for efficient and faithful task\-agnostic prompt compression\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 963–981\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.57),[Link](https://aclanthology.org/2024.findings-acl.57/)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1)\.
- R\. Y\. Pang, A\. Parrish, N\. Joshi, N\. Nangia, J\. Phang, A\. Chen, V\. Padmakumar, J\. Ma, J\. Thompson, H\. He, and S\. R\. Bowman \(2022\)QuALITY: question answering with long input texts, yes\!\.Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 5336–5358\.Cited by:[3rd item](https://arxiv.org/html/2606.04557#A11.I1.i3.p1.1),[3rd item](https://arxiv.org/html/2606.04557#S3.I1.i3.p1.1)\.
- B\. Peng, J\. Quesnelle, H\. Fan, and E\. Shippole \(2024\)YaRN: efficient context window extension of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 31932–31951\.Cited by:[§4](https://arxiv.org/html/2606.04557#S4.SS0.SSS0.Px4.p1.2)\.
- C\. V\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2025\)Scaling LLM Test\-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning\.InThe Thirteenth International Conference on Learning Representations,ICLR ’25,Singapore, Singapore\.External Links:[Link](https://openreview.net/forum?id=4FWAwZtd2n)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p1.1)\.
- J\. Strich, E\. K\. Isgorur, M\. Trescher, C\. Biemann, and M\. Semmann \(2026\)T2\-RAGBench: text\-and\-table benchmark for evaluating retrieval\-augmented generation\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 165–191\.External Links:[Link](https://aclanthology.org/2026.eacl-long.8/)Cited by:[4th item](https://arxiv.org/html/2606.04557#A11.I1.i4.p1.1),[Appendix H](https://arxiv.org/html/2606.04557#A8.p2.1),[4th item](https://arxiv.org/html/2606.04557#S3.I1.i4.p1.1)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§2\.1](https://arxiv.org/html/2606.04557#S2.SS1.SSS0.Px3.p2.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Wan, X\. Wu, Y\. Zhang, Y\. Xin, C\. Tao, Z\. Zhu, X\. Wang, S\. Luo, J\. Xiong, L\. Wang, and M\. Zhang \(2025\)D2O: Dynamic discriminative operations for efficient long\-context inference of large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HzBfoUdjHt)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang \(2024\)Model tells you where to merge: Adaptive KV cache merging for LLMs on long\-context tasks\.External Links:[Link](https://arxiv.org/abs/2407.08454),2407\.08454Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- M\. Weber, D\. Y\. Fu, Q\. Anthony, Y\. Oren, S\. Adams, A\. Alexandrov, X\. Lyu, H\. Nguyen, X\. Yao, V\. Adams, B\. Athiwaratkun, R\. Chalamala, K\. Chen, M\. Ryabinin, T\. Dao, P\. Liang, C\. Ré, I\. Rish, and C\. Zhang \(2024\)RedPajama: an open dataset for training large language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 116462–116492\.External Links:[Document](https://dx.doi.org/10.52202/079017-3697),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/d34497330b1fd6530f7afd86d0df9f76-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§4](https://arxiv.org/html/2606.04557#S4.SS0.SSS0.Px3.p3.3)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix A](https://arxiv.org/html/2606.04557#A1.p2.1),[Appendix K](https://arxiv.org/html/2606.04557#A11.SS0.SSS0.Px1.p3.1),[§3](https://arxiv.org/html/2606.04557#S3.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024\)CaM: cache merging for memory\-efficient LLMs inference\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://proceedings.mlr.press/v235/zhang24n.html)Cited by:[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f8-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.04557#S1.p3.1),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Zweiger, X\. Fu, H\. Guo, and Y\. Kim \(2026\)Fast KV compaction via attention matching\.arXiv preprint arXiv:2602\.16284\.Cited by:[Appendix H](https://arxiv.org/html/2606.04557#A8.p2.1),[§1](https://arxiv.org/html/2606.04557#S1.p4.2),[§6](https://arxiv.org/html/2606.04557#S6.SS0.SSS0.Px2.p1.1),[Model scale\.](https://arxiv.org/html/2606.04557#Sx2.SS0.SSS0.Px4.p1.1)\.

## Appendix AHyperparameters

Table[5](https://arxiv.org/html/2606.04557#A1.T5)summarizes the training hyper\-parameters used across all datasets\. All experiments use Qwen3\-8B666[https://huggingface\.co/Qwen/Qwen3\-8B](https://huggingface.co/Qwen/Qwen3-8B)as the base model with BF16 weights and FP32 Adam optimizer states\. Cartridge parameters are the only trainable parameters; all model weights remain frozen\. We train using a cluster with NVidia H200 and B200 cloud GPUs\.

We use Qwen3\-8B as released on Hugging Face\. The model card provided by the Qwen team documents training data, intended use, and limitationsYanget al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib22)\)\.

HyperparameterValueNotes*Optimization*OptimizerAdam \(FP32\)Moment tensors offloaded to NVMe for CPU\-resident cartsPeak learning rate \(η0\\eta\_\{0\}\)0\.05–0\.1Dataset\-dependent; 0\.1 for LongHealth, 0\.05 for othersLR scheduleLinear decay with warmupWarmup steps \(WW\)200Global warmupFinal LR multiplier \(αf\\alpha\_\{f\}\)0\.02ηmin=0\.02⋅η0\\eta\_\{\\text\{min\}\}=0\.02\\cdot\\eta\_\{0\}Warmup minimum LR0\.002Per\-cartridge warmup \(WcW\_\{c\}\)20 stepsApplied when a cartridge first enters the GPU poolTraining epochs80Global batch size128Packed sequence length8,192 tokens*Budget Manager & Rotation*Cartridge budget \(BB\)20Max cartridges on GPU simultaneouslyRotation interval \(RR\)10 stepsSteps between pool rotationsSwap fraction \(ϕ\\phi\)0\.5Fraction of pool replaced per rotationPrioritize least trainedYesCartridges with fewest steps swapped in firstEmpty cache interval5 rotationstorch\.cuda\.empty\_cache\(\)frequency*Cartridge Masking*Isolation probability \(PisoP\_\{\\text\{iso\}\}\)0\.75Prob\. of seeing only the relevant cartridgeMax active cartridges \(kmaxk\_\{\\text\{max\}\}\)10Max distractors visible per exampleMin active cartridges \(kmink\_\{\\text\{min\}\}\)1*Initialization & Padding*Init strategyCart\-specificEach cartridge initialized from its own documentCache paddingRepeatTiles existing KV vectors if doc<<cart sizeNum frozen tokens1BOS token frozen during training*Infrastructure*HardwareNVIDIA H200 & B200PrecisionBF16 \(weights\), FP32 \(optimizer\)Gradient checkpointingDisabledtorch\.compileDisabledTable 5:Training hyperparameters for the CAS framework\. Values shown are the defaults used across all five datasets unless noted otherwise\. The peak learning rate is the primary dataset\-dependent hyperparameter\.#### Dataset\-specific variations\.

The total number of optimizer steps varies by dataset due to different corpus sizes: LongHealth \(20 docs\) trains for∼12,480\{\\sim\}12\{,\}480steps, QuALITY \(115 docs\) for∼12,400\{\\sim\}12\{,\}400steps, QASPER \(407 docs\) for∼14,160\{\\sim\}14\{,\}160steps, and FinQA \(380 docs\) for∼14,160\{\\sim\}14\{,\}160steps\. The LR schedulemax\_stepsparameter is set accordingly\. For the isolation ablation in Table[3](https://arxiv.org/html/2606.04557#S4.T3), we usePiso=0\.75P\_\{\\text\{iso\}\}\{=\}0\.75for joint training andPiso=1\.0P\_\{\\text\{iso\}\}\{=\}1\.0for isolated training, with all 20 cartridges active simultaneously \(no rotation needed for 20 cartridges on a H100 GPU\)\.

#### Model Inference\.

For all experiments we evaluate the models with thinking mode enabled, temperature 0\.6, and 2048 maximum completion tokens\. We use a separate prompt for each dataset \(see Appendix[H](https://arxiv.org/html/2606.04557#A8)\), which we pass in the user turn\. In the RAG condition, the retrieved context replaces the full document in the prompt; in the Oracle text condition, the full document text is provided\. All reported results are averaged across at least 3 runs\.

## Appendix BEffective LR per Cartridge

Because cartridge rotation means each cartridge is active for only a fraction of total training steps, the effective number of gradient updates per cartridge is much smaller than the global step count\. A standard short\-decay schedule causes the learning rate to reach its minimum long before most cartridges have received sufficient updates, stalling their convergence\. We ablate this on FinQA by comparing a short\-decay schedule \(max\_steps=1600\\texttt\{max\\\_steps\}\{=\}1600\) against a slow\-decay schedule \(max\_steps=5000\\texttt\{max\\\_steps\}\{=\}5000,αf=0\.02\\alpha\_\{f\}\{=\}0\.02\) that maintains a higher effective learning rate throughout training \(Figure[5](https://arxiv.org/html/2606.04557#A2.F5)\)\. The short\-decay schedule reaches54\.8%54\.8\\%; switching to slow decay improves this to59\.0%59\.0\\%\(\+4\.2 points\)\. Adding per\-cartridge warmup and prioritize\-least\-trained scheduling further pushes accuracy to60\.8%60\.8\\%, as each cartridge receives a proper warmup phase when first loaded into the GPU budget\.

![Refer to caption](https://arxiv.org/html/2606.04557v1/images/lr_decay_ablation.png)Figure 5:LR schedule ablation on FinQA\. The slow\-decay schedule withmax\_steps=5000\\texttt\{max\\\_steps\}\{=\}5000and per\-cartridge warm\-up is superior tomax\_steps=1600\\texttt\{max\\\_steps\}\{=\}1600\.
## Appendix CImpact of Self\-Study Data Improvements

We ablate the cumulative effect of our self\-study data improvements on FinQA using 10 single\-doc cartridges of 512 tokens each\. We compare the three data generation strategies:

- •Original: The baseline procedure fromEyubogluet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib8)\)—one question per LLM call, using the target model \(Qwen3\-8B\) for both question and answer generation, with uniform random chunk sampling\.
- •DecoupledMQM\_\{Q\},n=20n\{=\}20: Multi\-question generation \(n=20n\{=\}20questions per call\) with a decoupled, more capable question model \(GPT\-OSS 120B\) while keeping the target model asMAM\_\{A\}\. Same uniform sampling\.
- •DecoupledMQM\_\{Q\},n=20n\{=\}20\+ prop\. to length: Same as above, but replacing uniform sampling with proportional\-to\-length document sampling, ensuring longer documents receive proportionally more training examples\.

Data StrategyAccuracy \(%\)Original55\.9DecoupledMQM\_\{Q\},n=20n\{=\}2062\.7DecoupledMQM\_\{Q\},n=20n\{=\}20\+ prop\. to length64\.7Oracle \(Text\)66\.8Table 6:Impact of self\-study data improvements on FinQA \(10 cartridges, 512 tokens/cart\)\.The results \(Table[6](https://arxiv.org/html/2606.04557#A3.T6)\) show that decoupling the question model and batching 20 questions per call provides the largest single improvement \(\+6\.8 points over the original\), by generating more diverse and higher\-quality questions that better cover the document’s factual content\. Adding proportional\-to\-length sampling contributes a further \+2\.0 points by ensuring that longer, information\-dense documents receive training signal commensurate with their expected greater complexity\.

## Appendix DScoring Metrics

We use dataset\-specific scoring metrics:

- •T2\-RAGBench/FinQA: Formula execution accuracy\. The predicted formula is parsed and executed; the numerical result is compared to the gold answer with a relative tolerance of 1%\.
- •LongHealth: Fuzzy option matching\. The predicted answer is compared against all five options using string similarity, and the closest match is selected\.
- •QASPER: Average token F1 with multi\-reference support, following the standard QASPER evaluation protocol\. For each question, we compute the token\-level F1 against all available reference answers and take the maximum\.
- •QuALITY: Exact match on the extracted answer letter \(A, B, C, or D\)\.
- •TechQA: LLM\-as\-a\-Judge\. A separate judge model \(DeepSeek\-Distilled\-Qwen\-32B\) evaluates whether the generated answer is factually equivalent to the reference answer\. The judge outputs a JSON object with a justification and a binary grade\. A score of 1 is assigned when the judge returns"grade": "correct", and 0 otherwise\. Table[7](https://arxiv.org/html/2606.04557#A4.T7)shows the exact judge prompt\. RolePromptSystemYou are to act as an impartial judge, evaluating whether an answer to a question matches a provided reference answer\. Your task is to determine if the given answer is correct based on its factual equivalence to the reference answer, ignoring differences in punctuation and phrasing\.When evaluating the answer, consider the following criteria:1\. Factual equivalence: Does the answer convey the same information as the reference answer?2\. Completeness: Does the answer address all parts of the question that the reference answer addresses?3\. Accuracy: Is the information in the answer consistent with the reference answer?4\. Additional information: If the answer contains more information than the reference answer, does it remain consistent and not contradict itself?Your response should be structured as follows:1\. A justification for your decision, explaining your reasoning based on the evaluation criteria\.2\. A grade of either "correct" or "incorrect"\.Important note on deflections and invalid questions:\- If the answer is a deflection or does not attempt to answer the question, grade it as "incorrect" unless the reference answer is also a deflection\.\- If both the answer and the reference answer indicate that the question is invalid or cannot be answered, grade it as "correct"\.\- If the answer is a placeholder like \{\{YOUR\_ANSWER\}\}, any generic phrase that does NOT address the question, or an empty answer, then count it as "incorrect"\.Your response should be in json format as follows:\{"justification": "\.\.\.", "grade": "correct" or "incorrect"\}UserHere is the question:\{query\}Here is the answer to be judged:\{answer\}Here is the reference answer:\{expected\_answer\}Table 7:LLM\-as\-a\-Judge prompt used for scoring free\-form answers on TechQA\. The judge model \(DeepSeek\-Distilled\-Qwen\-32B\) receives the system prompt defining evaluation criteria and a user message containing the question, generated answer, and reference answer\.

For all datasets, we report accuracy as the percentage of correctly answered questions\.

## Appendix ECartridge Size Statistics

Because each cartridge is sized proportionally to its source document, the actual number of KV tokens per cartridge varies across documents within a dataset\. Table[8](https://arxiv.org/html/2606.04557#A5.T8)reports the minimum, mean, and maximum cartridge sizes \(in tokens\) for each dataset at each target compression ratio used in our experiments\.

All cartridge sizes are rounded up to the nearest multiple of 16 tokens\. This constraint arises from the paged KV\-cache implementation in our inference server \(tokasaurus777[https://github\.com/ScalingIntelligence/tokasaurus](https://github.com/ScalingIntelligence/tokasaurus)\), which manages memory in fixed\-size pages of 16 tokens\. Aligning cartridge sizes to page boundaries avoids internal fragmentation and ensures that each cartridge occupies an integer number of pages, enabling efficient memory allocation and deallocation during both training \(pool rotation\) and inference \(cartridge loading/unloading\)\. We additionally enforce a minimum cartridge size of 16 tokens \(one page\), ensuring that even the shortest documents receive at least one full page of KV representation\.

DatasetComp\.MinMeanMaxTotalLongHealth \(20 docs\)2×2\\times5,0405,8596,640117,1845×5\\times2,0162,3492,65646,97610×10\\times1,0081,1781,32823,56820×20\\times51259367211,85650×50\\times2082422724,848100×100\\times1121261442,512QASPER \(407 docs\)2×2\\times4642,3494,912955,9683×3\\times3041,5693,280638,4005×5\\times1929441,968384,33610×10\\times96476992193,90420×20\\times6424249698,64050×50\\times6410320842,016100×100\\times646811227,568QuALITY \(115 docs\)2×2\\times1,1362,8654,208329,4565×5\\times4641,1511,696132,33610×10\\times24057984866,54420×20\\times12829343233,74450×50\\times6412317614,176100×100\\times6474968,464FinQA \(380 docs\)2×2\\times1125241,280199,1845×5\\times6421451281,45610×10\\times6411225642,52820×20\\times646912826,09650×50\\times16296411,088100×100\\times1617326,448TechQA \(471 docs\)2×2\\times644602,048216,8325×5\\times641931,00890,78410×10\\times6410951251,24820×20\\times647525635,32850×50\\times646511230,496100×100\\times1619648,752Table 8:Cartridge size statistics \(in tokens\) per dataset and target compression ratio\.Min: smallest cartridge in the collection\.Mean: average tokens per cartridge\.Max: largest cartridge\.Total: sum of all cartridge sizes\. All sizes are multiples of 16 \(the page size\)\.At high compression ratios \(50×50\\times–100×100\\times\), the shortest documents in FinQA and TechQA are clamped to the 16\-token minimum\. For example, a 500\-token FinQA document at100×100\\timescompression would nominally require only 5 tokens, but is rounded up to 16\. This means the*effective*compression ratio for short documents is lower than the target—a 500\-token document with a 16\-token cartridge achieves31×31\\timesrather than100×100\\times\. Conversely, the longest documents in each collection achieve compression ratios close to the target\. The wide min–max range within a dataset \(e\.g\., 64–2,048 for TechQA at2×2\\times\) reflects the natural variation in document lengths: the proportional sizing ensures that each document receives a KV budget commensurate with its information content\.

## Appendix FRAG Baseline: Indexing and Retrieval Details

We describe the full RAG pipeline below\.

#### Document indexing\.

Each dataset’s documents are indexed with chunk sizeC∈\{128,256,512,1024\}C\\in\\\{128,256,512,1024\\\}tokens\. Documents are chunked using a fixed\-size strategy with a 10% token overlap between adjacent chunks\. Chunks are embedded using Amazon Titan Embed Text v2 \(amazon\.titan\-embed\-text\-v2:0\) via Amazon Bedrock888[https://aws\.amazon\.com/bedrock](https://aws.amazon.com/bedrock)with 1024\-dimensional embeddings and retrieved via cosine similarity\.

#### Retrieval\.

At evaluation time, each question is issued as a retrieval query against the Knowledge Base\. We retrieve the top\-KKchunks forK∈\{1,3,5,10\}K\\in\\\{1,3,5,10\\\}\. To avoid redundant API calls, we retrieveK=10K=10once per chunk size and slice for smallerKKvalues\. Retrieved chunks are concatenated in score order and prepended to the question as the context for the reader model\.

#### Hyperparameter selection\.

We report the best result across all 16 chunk/top\-KKcombinations \(4×44\\times 4\) for each dataset as the “RAG best” entry in Table[2](https://arxiv.org/html/2606.04557#S3.T2)\. The optimal configuration varies by dataset: datasets with short, self\-contained answers \(TechQA, FinQA\) benefit from larger chunks and higherKK, while datasets requiring precise span extraction \(QASPER\) are less sensitive to retrieval configuration\.

#### Effective token counts\.

Table[10](https://arxiv.org/html/2606.04557#A6.T10)reports the average number of context tokens consumed per query for Text RAG at chunk size 1024 across differentkkvalues\. Due to the 10% overlap between adjacent chunks and shorter final chunks in some documents, the effective token count is slightly below the nominalk×1024k\\times 1024\.

#### Unique cartridges per query\.

Because multiple retrieved chunks may originate from the same source document, the number of unique documents loaded in Cartridge RAG is often lower than the number of retrieved chunkskk\. Table[9](https://arxiv.org/html/2606.04557#A6.T9)reports the average number of unique documents covered by the top\-kkchunks \(at chunk size 1024\)\. Datasets with short documents \(FinQA, TechQA\) exhibit high chunk diversity—10 chunks span 8–9 distinct documents—while datasets with longer documents \(LongHealth, QuALITY\) show more within\-document clustering, with 10 chunks covering only 4–5 documents on average\.

Datasetk=1k\{=\}1k=3k\{=\}3k=5k\{=\}5k=10k\{=\}10LongHealth1\.01\.82\.54\.4QuALITY1\.01\.42\.24\.8FinQA1\.03\.05\.08\.7TechQA1\.03\.04\.98\.1Table 9:Average number of unique source documents covered by the top\-kkretrieved chunks \(chunk size 1024\)\. Datasets with shorter documents \(FinQA, TechQA\) have higher chunk diversity, while longer\-document datasets \(LongHealth, QuALITY\) exhibit within\-document clustering\.Datasetk=1k\{=\}1k=3k\{=\}3k=5k\{=\}5k=10k\{=\}10LongHealth9862,9604,9449,860QuALITY9412,8564,7339,412FinQA∼\{\\sim\}960∼\{\\sim\}2,880∼\{\\sim\}4,800∼\{\\sim\}9,600TechQA∼\{\\sim\}960∼\{\\sim\}2,880∼\{\\sim\}4,800∼\{\\sim\}9,600Table 10:Average context tokens per query for Text RAG \(chunk size 1024\)\.

## Appendix GRetrieval Quality: Recall and MRR

To contextualize the Cartridge RAG results in §[4](https://arxiv.org/html/2606.04557#S4)*Cartridge RAG*, we report document\-level retrieval quality of the dense retriever used by both the Text RAG and Cartridge RAG pipelines\. For each query we retrieve1010chunks, deduplicate to unique source documents, and compute Recall@KKand MRR@KKover the resulting ranked document list \(soKKcounts unique documents, not chunks\)\. Tables[11](https://arxiv.org/html/2606.04557#A7.T11)–[14](https://arxiv.org/html/2606.04557#A7.T14)report these metrics for the four chunk sizes used in our sweep\.

ChunkR@1R@3R@5R@10MRR@1MRR@3MRR@5MRR@1012879\.288\.591\.295\.879\.283\.684\.585\.525676\.587\.591\.295\.576\.581\.582\.583\.351276\.089\.093\.097\.076\.081\.882\.883\.6102476\.586\.591\.296\.876\.581\.082\.483\.4

Table 11:LongHealth document\-level retrieval \(2020docs,400400queries\)\.ChunkR@1R@3R@5R@10MRR@1MRR@3MRR@5MRR@1012885\.391\.193\.195\.285\.388\.088\.588\.925688\.092\.694\.095\.488\.090\.190\.590\.851288\.392\.193\.295\.588\.390\.190\.490\.9102487\.191\.493\.194\.887\.189\.189\.690\.0

Table 12:QuALITY document\-level retrieval \(115115docs,2,0862\{,\}086queries\)\.LongHealth and QuALITY have nearly saturated retrieval \(≥95%\\geq 95\\%Recall@10\), so any gap between Text RAG and Oracle \(Text\) on these datasets is attributable to information loss in the chunked text rather than retrieval failures\.

ChunkR@1R@3R@5R@10MRR@1MRR@3MRR@5MRR@1012870\.485\.790\.795\.970\.477\.078\.079\.025670\.587\.391\.896\.170\.578\.079\.079\.051269\.286\.091\.595\.669\.277\.078\.079\.0102469\.287\.091\.896\.069\.277\.078\.079\.0

Table 13:FinQA document\-level retrieval \(380380docs,1,1471\{,\}147queries\)\.ChunkR@1R@3R@5R@10MRR@1MRR@3MRR@5MRR@1012879\.889\.091\.893\.179\.883\.884\.484\.625680\.389\.792\.693\.980\.384\.585\.285\.451282\.390\.293\.195\.282\.385\.886\.586\.9102483\.192\.193\.895\.283\.187\.387\.787\.9

Table 14:TechQA document\-level retrieval \(496496docs,610610queries\)\.FinQA and TechQA both exhibit high retrieval quality \(≥95%\\geq 95\\%Recall@10\), comparable to LongHealth and QuALITY\. FinQA documents contain distinctive company names and financial terminology that make retrieval effective despite the large corpus \(380 docs\)\. TechQA’s IBM Technotes cover distinct products and error codes, yielding strong R@1 \(8080–83%83\\%\) that improves with larger chunk sizes as more diagnostic context is captured per chunk\.

## Appendix HEvaluation Prompts

We use task\-specific system prompts during evaluation\. All prompts instruct the model to wrap its final answer in<answer\>tags\. In the Cartridge setting, the document context is encoded in the trained KV cache prefix and is not included in the text prompt—the model receives only the system prompt and the user question\. In the Oracle \(Text\) baseline, the full document text is prepended to the user message\. Table[15](https://arxiv.org/html/2606.04557#A8.T15)lists the exact prompts used\.

BenchmarkSystem Prompt \+ User Message FormatT2\-RAGBench / FinQASystem: You are an expert in answering financial questions by constructing mathematical formulas based on a simple syntax\.\- Task: Provide a FORMULA ANSWER to the question based on the given context\.Guidelines:1\. Answer Type: A formula is either a number or one of: add\(f1, f2\), subtract\(f1, f2\), multiply\(f1, f2\), divide\(f1, f2\), exp\(f1, f2\), greater\(f1, f2\)2\. Reasoning: Carefully analyze the context\. Pay special attention to the table\.3\. Final Answer: "<answer\> FORMULA </answer\>"User: \{context\}Question: \{question\}LongHealthSystem: Please reference the patient medical records to answer the user’s questions\. Choose the single best option and provide your answer exactly as it appears in the options\.Wrap your answer in: <answer\> The correct option text here </answer\>User: \{question\}A\. \{option\_a\} B\. \{option\_b\} C\. \{option\_c\} D\. \{option\_d\} E\. \{option\_e\}QASPERSystem: You are a research assistant answering questions about a scientific paper\. Answer as briefly as possible\. Give only the answer, no explanation\. If the question cannot be answered from the paper, say "Unanswerable"\. For yes/no questions, answer "Yes" or "No"\. Wrap your answer in: <answer\> \.\.\. </answer\>User: \{question\}QuALITYSystem: You are a careful reader answering multiple\-choice questions about a long article\. Read the article and choose the single best answer option\. Provide your answer as the letter \(A, B, C, or D\) wrapped in answer tags\. Example: <answer\> B </answer\>User: \{question\}A\. \{option\_a\} B\. \{option\_b\} C\. \{option\_c\} D\. \{option\_d\}Answer with the letter of the correct option \(A, B, C, or D\)\.TechQASystem: You are an expert technical support assistant specializing in IT infrastructure, software products, and enterprise systems\. Answer the technical question to the best of your ability\. Be concise and factual\. Wrap your answer in: <answer\> YOUR\_ANSWER </answer\>User: \{question\}Table 15:Evaluation prompts used for each benchmark\. Variables in braces are replaced with benchmark data at evaluation time\.
## Appendix ISelf\-Study Data Synthesis Details

#### Question generation model\.

We use GPT\-OSS 120BAgarwalet al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib23)\)as the question generator \(MQM\_\{Q\}\)\. This model receives the full document context in its system prompt and generates batches of 20 diverse questions per call\. The answer generator \(MAM\_\{A\}\) is the target model itself \(Qwen3\-8B\), ensuring that the distillation signal reflects the student’s own output distribution\.

#### Seed prompt types\.

Each synthesis call randomly selects a seed prompt type from five categories:*structuring*\(requests to organize information into JSON, YAML, or other formats\),*summarization*\(requests to summarize specific sections\),*question*\(factual recall and reasoning questions\),*use\_case*\(practical downstream application tasks\), and*creative*\(open\-ended discussion prompts\)\. This diversity ensures the cartridge is trained on varied interaction patterns rather than only factoid QA\.

#### Multi\-question prompt format\.

In batched mode, the question generator receives the following instruction \(shown for the*question*type\):

> Generate \{n\} diverse questions that test knowledge of the information in the corpus above\. Each question should cover a different fact, detail, or aspect of the corpus\. Vary the style: mix factual recall, comparison, reasoning, and detail\-oriented questions\. Include specific details \(ids, names, titles, dates, numerical values, etc\.\) in each question so it is clear what you are asking about\. Output ONLY a JSON array of strings, e\.g\. \["question 1", "question 2", \.\.\.\]\. No other text, no markdown fences, no explanation\.

The structured JSON output format enables reliable parsing; responses that fail to parse are discarded \(typically<5%\{<\}5\\%of calls\)\.

#### Sampling rounds and temperatures\.

For each dataset, we run 4 independent sampling rounds, one per seed prompt type \(question, structuring, summarization, use\_case\), each generating 10,000 samples for a total of 40,000 training examples per dataset\.121212We exclude the*creative*seed type from training data as it produces open\-ended prompts less suited for distillation\.MQM\_\{Q\}\(GPT\-OSS 120B, the question generator\) uses temperature0\.60\.6with top\-p=0\.95p\{=\}0\.95, top\-k=20k\{=\}20, and a maximum of 4,096 completion tokens per call\.MAM\_\{A\}\(Qwen3\-8B, the teacher/answer generator\) uses temperature0\.00\.0\(greedy decoding\) with a maximum of 2,048 completion tokens, ensuring deterministic distillation targets\. Both models operate with thinking mode enabled\.

#### Sampling configuration\.

Within each round, we generate 10,000 synthesis samples with a batch size of 4 contexts and up to 128 parallel API calls\. Documents are sampled using the proportional\-to\-length strategy described in §[2\.3](https://arxiv.org/html/2606.04557#S2.SS3), with a fixed random seed for reproducibility\. For multi\-note documents \(e\.g\., LongHealth patient records\), we sample one note per prompt to ensure fine\-grained coverage of individual clinical notes\.

## Appendix JSelf\-Study Data Quality Analysis

To understand the failure modes of the original Self\-Study procedure, we conduct a detailed analysis on 9 FinQA cartridges \(512 tokens each, trained for 80 epochs\)\. We measure*value coverage*: for each unique number in the original document, we count the fraction of training samples that reproduce it\.

#### Synthesizer bias toward narrative over tables\.

The synthesizer preferentially generates questions about prose commentary rather than specific table cell values\. Table[16](https://arxiv.org/html/2606.04557#A10.T16)shows that across all 9 cartridges, only 41% of unique numbers in the training data are grounded in the original document\. The remaining 59% are derived calculations \(62%\), rounded/reformatted values \(22%\), or pure fabrications \(15%\)\.

CategoryCount%Grounded \(in document\)28141%Derived calculations25537%Rounded/reformatted9013%Pure fabrications629%Total not in document40759%Table 16:Classification of unique numbers appearing in Self\-Study training data across 9 FinQA cartridges \(688 unique numbers total, excluding years\)\.For example, in a Lockheed Martin document, the text\-derived delta “operating profit decreased $264 million” achieves 87% coverage \(80/92 samples\), while the underlying table values it was computed from—$6,608M, $6,770M, $7,092M \(net sales 2014–2016\)—appear at only 3–5% coverage\. All 4 test questions requiring these table values score 0%\.

#### Does coverage predicts correctness?

Table[17](https://arxiv.org/html/2606.04557#A10.T17)shows the relationship between minimum value coverage and answer correctness\. A coverage threshold of∼25%\{\\sim\}25\\%is necessary \(87% of correct answers meet it\) but not sufficient \(46% of incorrect answers also meet it\), indicating that additional failure modes—such as entity confusion—contribute to errors even when values are present\.

Min coverageCorrect \(n=15n\{=\}15\)Incorrect \(n=13n\{=\}13\)≥\\geq0%100%100%≥\\geq10%100%69%≥\\geq25%87%46%≥\\geq30%67%23%≥\\geq50%27%23%Table 17:Fraction of questions meeting a minimum coverage threshold, split by correctness\. Coverage≥25%\\geq 25\\%is necessary but not sufficient\.
#### Entity confusion\.

The cartridge stores numeric values but loses entity\-value associations\. In a UPS shareowner return document, the model correctly retrieves UPS’s return \($121\.46, 100% coverage\) but substitutes the S&P 500 value \($108\.59, 69% coverage\) for the Dow Jones Transportation value \($127\.07, 69% coverage\) when computing their difference—both distractor values have identical coverage, and the model picks the wrong entity\.

#### Hallucination despite self\-verification\.

The model fabricates plausible values, attempts chain\-of\-thought verification, detects inconsistencies, and proceeds anyway\. In a Hologic acquisition document, the model needs the goodwill value \($6,900, 28% coverage\) but produces three different hallucinated values across 3 runs \(4,200, 6,000, 3,000\)\. In each run, it sums the allocation components, notices the total does not match $31,300, yet outputs the wrong formula regardless\.

## Appendix KArtifact Licenses and Terms of Use

#### Licenses and terms for use/distribution of artifacts\.

The five benchmarks used in this work are publicly available under the following licenses:

- •LongHealthAdamset al\.\([2024](https://arxiv.org/html/2606.04557#bib.bib14)\): Released under the Apache\-2\.0 License\.131313[https://github\.com/kbressem/LongHealth/blob/main/LICENSE](https://github.com/kbressem/LongHealth/blob/main/LICENSE)The dataset consists of entirely fictional patient records created by the authors; no real patient data is included\.
- •QASPERDasigiet al\.\([2021](https://arxiv.org/html/2606.04557#bib.bib15)\): Released under the CC BY 4\.0 License\.141414[https://huggingface\.co/datasets/allenai/qasper](https://huggingface.co/datasets/allenai/qasper)The dataset consists of questions and answers over NLP research papers; the dataset itself is distributed under CC BY 4\.0\.
- •QuALITYPanget al\.\([2022](https://arxiv.org/html/2606.04557#bib.bib16)\): Released under the CC BY 4\.0 License\.151515[https://nyu\-mll\.github\.io/quality/](https://nyu-mll.github.io/quality/)Source texts are drawn from Project Gutenberg \(public domain\) and other permissively licensed collections, including nonfiction and fiction sources such as Slate articles from the Open American National Corpus, The Long\+Short, Freesouls, and Open Access books\. These texts are published works and do not contain personal data, though some may include mature themes typical of literary and journalistic content\.
- •FinQAChenet al\.\([2021](https://arxiv.org/html/2606.04557#bib.bib12)\): Released under the MIT License\.161616[https://github\.com/czyssrs/FinQA/blob/main/LICENSE](https://github.com/czyssrs/FinQA/blob/main/LICENSE)The underlying financial reports are sourced from corporate filings made available via the SEC EDGAR system, including earnings reports and annual/quarterly reports \(e\.g\., 10\-K and 10\-Q documents\)\. These are publicly accessible financial disclosures prepared by reporting companies\. T2\-RAGBenchStrichet al\.\([2026](https://arxiv.org/html/2606.04557#bib.bib13)\)is also released under the MIT License\.
- •

The base model used in our experiments,Qwen3\-8BYanget al\.\([2025](https://arxiv.org/html/2606.04557#bib.bib22)\), is released under the Apache 2\.0 License, which permits research and commercial use\.

We do not release new datasets in this work\. The trained cartridge artifacts \(KV cache parameters\) are derivatives of the above datasets and the Qwen3\-8B model weights; any release of such artifacts would be subject to the intersection of the applicable licenses above\.

#### Offensive content and personal data\.

- •LongHealth: The dataset consists of entirely fictional patient records with no real individuals\. No anonymization was required\. We verified that no real names, addresses, or identifying information appear in the data\.
- •QASPER: Source texts are NLP research papers\. No personal data or offensive content is expected; the domain is scientific writing\.
- •QuALITY: Source texts are fiction and non\-fiction narratives, these are published literary works; no personal data is present\.
- •FinQA / T2\-RAGBench: Source texts are corporate earnings reports from SEC EDGAR\. These are formal financial documents; no personal data or offensive content is present\.
- •TechQA: Source texts are IBM Technote IT support documents\. These are technical documentation; no personal data or offensive content is expected\.

No additional anonymization steps were taken beyond those applied by the original dataset creators, as none of the datasets contain personal data about private individuals\.

## Appendix LAI use disclosure

We used AI to assist with code writing and manuscript typesetting\.