Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

# A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory
Source: [https://arxiv.org/html/2607.28263](https://arxiv.org/html/2607.28263)
Hanzuo Liu†‡Xuan Qi†∗Chunyu Liu†∗Haotian Zhong†∗ Yulong Wang‡Rayying‡Key‡Alex Lamb†Mingyu Gao† †Tsinghua University‡Tencent \{lhz24@mails\., gaomy@\}tsinghua\.edu\.cn [https://github\.com/liuhanzuo/COMem](https://github.com/liuhanzuo/COMem)

###### Abstract

Transformer layers appear to play different roles: semantic information becomes accessible in lower and middle layers, whereas upper layers increasingly specialize representations for prediction\. We turn this division of labor intoCoMem\(Comprehension Memory\), which writes each context chunk only through an intermediate layer, retrieves a fixed number of cached residual tensor, and recomputes the query\-conditioned upper layers over the resulting pack\. For a fixed retrieval budget, model\-side read compute and memory are independent of stored\-context length\. We evaluate a continued\-trained Qwen3\-8B base LM under a unified chat\-template\-free protocol\. The backbone is frozen; the flagship trains only a rank\-32 self\-distillation LoRA on plain PG\-19, and we report an adapter\-free arm separately\. CoMem reaches 97\.05 on RULER and 38\.27 on LoCoMo versus 34\.59 for full\-context KV\-Direct; the dialogue\-memory advantage survives conversation\-cluster resampling and an independent judge\. Results on additional long\-context and long\-document tasks expose both the benefits of bounded retrieval and its in\-window compression tax\. Controlled depth sweeps show that deeper caching lowers per\-query recomputation but incurs a fidelity loss that self\-distillation substantially repairs\. In a separate adapter\-free efficiency control on an NVIDIA H20 at 128k, CoMem uses 18\.26 GB rather than 89\.36 GB and achieves a7\.83×7\.83\\timesprefill speedup\. These results suggest that long\-context memory can be organized along the layer axis, not only the token axis\.

Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded\-Context Memory

A Preprint

Hanzuo Liu†‡Xuan Qi†∗Chunyu Liu†∗Haotian Zhong†∗Yulong Wang‡Rayying‡Key‡Alex Lamb†Mingyu Gao††Tsinghua University‡Tencent\{lhz24@mails\., gaomy@\}tsinghua\.edu\.cn[https://github\.com/liuhanzuo/COMem](https://github.com/liuhanzuo/COMem)
††footnotetext:Anonymous repository:[https://anonymous\.4open\.science/r/COMem\-Anonymous/](https://anonymous.4open.science/r/COMem-Anonymous/)## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.28263v1/x1.png)Figure 1:CoMem in one picture\.Semantic information becomes usable by mid\-depth, so CoMem caches the per\-token residual tensor at depthjjand recomputes the upper layers on a fixed retrieved pack\. For fixedkk, chunk size, and query length, the model\-side online read working set is independent of stored\-context length\. The accuracy curve uses the distilled flagship, whereas the7\.83×7\.83\\times/18\.26 GB inset is a separate adapter\-free efficiency control\. CoMem remains accurate beyond the unextended backbone’s native window; we do not compare against a YaRN\-extended full\-context model\.Long\-context language modeling incurs quadratic prefill cost from self\-attention and linear memory growth from the key–value \(KV\) cache\.\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.28263#bib.bib33); Kwonet al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib38)\)\. It also inherits the backbone’s position range: Qwen3\-8B\(Yang and others,[2025](https://arxiv.org/html/2607.28263#bib.bib31)\)has a40,96040\{,\}960\-token native window \(the configured131,072131\{,\}072limit requires inactive YaRN scaling\(Penget al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib36)\)\), so we treat longer full\-context runs only as unextended stress references\.

Most memory\-efficient methods evict or compress KV states\(Xiao and others,[2024b](https://arxiv.org/html/2607.28263#bib.bib6); Zhang and others,[2023](https://arxiv.org/html/2607.28263#bib.bib7); Li and others,[2024](https://arxiv.org/html/2607.28263#bib.bib8); Cai and others,[2024](https://arxiv.org/html/2607.28263#bib.bib15); Liuet al\.,[2024a](https://arxiv.org/html/2607.28263#bib.bib27)\), retrieve relevant text or memory units\(Lewiset al\.,[2020](https://arxiv.org/html/2607.28263#bib.bib39); Xiao and others,[2024a](https://arxiv.org/html/2607.28263#bib.bib5); Tanget al\.,[2024](https://arxiv.org/html/2607.28263#bib.bib28); Qiuet al\.,[2025b](https://arxiv.org/html/2607.28263#bib.bib42)\), or write into a trained memory\(Wang and others,[2024](https://arxiv.org/html/2607.28263#bib.bib22); Wanget al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib26); Qiuet al\.,[2025a](https://arxiv.org/html/2607.28263#bib.bib41)\)\. Residual\-stream reuse can reconstruct KV states without storing them independently\(Qasimet al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib24)\)\. CoMem combines bounded token selection with*depth\-partitioned computation reuse*: it asks both which chunks to retrieve and how much of their computation can be reused\. The premise is that lower and middle layers make semantic information accessible, whereas upper layers increasingly specialize it for the current query and prediction\(Ladet al\.,[2024](https://arxiv.org/html/2607.28263#bib.bib25)\)\.

We validate this depth division with probes, causal truncation, and split\-depth sweeps \(§[3](https://arxiv.org/html/2607.28263#S3)\), then turn it intoCoMem\(Figure[1](https://arxiv.org/html/2607.28263#S1.F1)\)\. DuringWrite, each chunk is processed only to depthjjand its residual tensor is cached\. DuringRead, a bounded set of relevant tensors is packed with the query and only layers\[j:L\]\[j\{:\}L\]are recomputed\. The external store remains linear in stored context, but for fixed retrieval breadth the model\-side online read does not\. The flagship uses a rank\-32 self\-distillation LoRA\(Hintonet al\.,[2015](https://arxiv.org/html/2607.28263#bib.bib40); Huet al\.,[2022](https://arxiv.org/html/2607.28263#bib.bib37)\)on a frozen backbone; an adapter\-free arm isolates the architecture\.

##### Contributions\.

- •Depth as a measurable reuse axis\.Semantic content becomes accessible near mid\-depth, while the zero\-shot readable boundary depends on model scale and can be shifted by lightweight adaptation\.
- •A bounded depth\-partitioned memory\.CoMem caches one residual state per token, exactly resumes the upper transformer, and combines bounded retrieval with full cross\-chunk attention\. On Qwen3\-8B the cached state is1/181/18the bytes of full bf16 KV per stored token\.
- •Controlled quality and efficiency evidence\.Under a single chat\-template\-free Qwen3\-8B evaluation protocol , CoMem reaches 97\.05 on RULER and 38\.27 on LoCoMo versus 34\.59 for KV\-Direct; the dialogue gain survives cluster resampling and an independent judge\. A separate adapter\-free H20 control at 128k uses 18\.26 GB instead of 89\.36 GB and achieves a7\.83×7\.83\\timesfull\-write\-inclusive prefill speedup\. Ablations isolate retrieval, split depth, cross\-chunk interaction, and distillation\.

## 2Related Work

We organize prior work along the same three axes that structure our design \(§[3](https://arxiv.org/html/2607.28263#S3)\):*what*to cache,*how deep*/ how much to train, and*how*to read it back\.

##### What to cache: full\-depth KV vs\. one intermediate residual tensor\.

The dominant approach to long context bounds the key–value \(KV\) cache along the*token*axis, retaining a compressed subset of*full\-depth*KV\. StreamingLLM\(Xiao and others,[2024b](https://arxiv.org/html/2607.28263#bib.bib6)\)keeps a recent window plus attention\-sink tokens; H2O\(Zhang and others,[2023](https://arxiv.org/html/2607.28263#bib.bib7)\)and SnapKV\(Li and others,[2024](https://arxiv.org/html/2607.28263#bib.bib8)\)evict all but heavy\-hitter or predicted\-attended positions; PyramidKV\(Cai and others,[2024](https://arxiv.org/html/2607.28263#bib.bib15)\)and LCKV\(Wu and Tu,[2024](https://arxiv.org/html/2607.28263#bib.bib16)\)reallocate or share the budget*across*layers; and MiniCache\(Liuet al\.,[2024a](https://arxiv.org/html/2607.28263#bib.bib27)\)merges redundant KVs across neighboring layers\. CompressKV\(Linet al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib3)\)adds a retrieval\-head eviction criterion\. Closest to us, HCache\(Gaoet al\.,[2025](https://arxiv.org/html/2607.28263#bib.bib1)\)caches a mid\-layer activation and recomputes the layers above it—but post\-hoc, untrained, and without retrieval—and KV\-Direct\(Qasimet al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib24)\)shows the KV cache is a deterministic projection of the residual stream and recomputes the*full\-depth*KV exactly, yet keeps every token so its working set still grows with context\. Most token\-axis methods retain selected full\-depth KV, whereas HCache is the closest mid\-depth\-recompute design but lacks relevance selection\. Following the depth division of labor \(§[3](https://arxiv.org/html/2607.28263#S3),*what to cache*\) and the depth\-semantics rationale ofLadet al\.\([2024](https://arxiv.org/html/2607.28263#bib.bib25)\), CoMem instead caches one depth\-jjresidual tensor per chunk \(one vector per token\) and reconstructs the upper layers on demand\.

##### How deep: fixed or trained memory vs\. a tunable depth split\.

A second line fixes the memory by architecture or by training\. MemoryLLM\(Wang and others,[2024](https://arxiv.org/html/2607.28263#bib.bib22)\)maintains a fixed\-size latent pool self\-updated inside the transformer \(we include its released Llama\-3 checkpoint as an out\-of\-backbone reference\), LongMem\(Wanget al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib26)\)retrieves external states through a trained side network, and RecursiveSummarizing\(Wang and others,[2023](https://arxiv.org/html/2607.28263#bib.bib23)\)keeps a natural\-language summary\. YOCO\(Sun and others,[2024](https://arxiv.org/html/2607.28263#bib.bib4)\)shares one global KV in a redesigned decoder–decoder, while CEPE\(Yenet al\.,[2024](https://arxiv.org/html/2607.28263#bib.bib30)\)adds a parallel context encoder and cross\-attention\. Others induce compressibility through continued training: Activation Beacon\(Zhang and others,[2024](https://arxiv.org/html/2607.28263#bib.bib17)\)learns to compress per\-layer activations into beacon tokens, and KV\-CAT\(Gelberget al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib2)\)trains for cache compressibility on the token/slot axis\. CoMem instead partitions on the*depth*axis and leaves the backbone frozen\. Its inference\-time memory operations require no parameter updates, while the evaluated flagship trains only a lightweight self\-distilled LoRA; an adapter\-free variant is reported separately\. CoMem exposes the splitjjas a single RAG↔\\leftrightarrowclosed\-book knob \(§[3](https://arxiv.org/html/2607.28263#S3),*how deep*\)\.

##### How to read: retrieval\-integrated memory vs\. external recompute\.

Retrieval\-augmented generation supplies selected text to a model\(Lewiset al\.,[2020](https://arxiv.org/html/2607.28263#bib.bib39)\)\. Within long\-context architectures, Landmark Attention\(Mohtashami and Jaggi,[2023](https://arxiv.org/html/2607.28263#bib.bib18)\)selects blocks through learned landmark tokens; FoT/LongLLaMA\(Tworkowski and others,[2023](https://arxiv.org/html/2607.28263#bib.bib19)\)uses an external contrastively trained\(k,v\)\(k,v\)memory; and InfLLM\(Xiao and others,[2024a](https://arxiv.org/html/2607.28263#bib.bib5)\)looks up token\-relevant memory units inside attention\. Quest\(Tanget al\.,[2024](https://arxiv.org/html/2607.28263#bib.bib28)\)and RetrievalAttention\(Liuet al\.,[2024b](https://arxiv.org/html/2607.28263#bib.bib29)\)instead retrieve query\-relevant KV pages or vectors during attention\. Serving systems such as CacheBlend\(Yao and others,[2024](https://arxiv.org/html/2607.28263#bib.bib20)\)and RAGCache\(Jin and others,[2024](https://arxiv.org/html/2607.28263#bib.bib21)\)precompute or reuse KV states for retrieved chunks\. These methods differ in their stored unit and training requirements, but retrieval is coupled to token/KV memory or the attention path\. CoMem keeps selection external: iterative BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.28263#bib.bib34)\)chooses chunk residual tensors, after which the upper layers are recomputed with full cross\-chunk attention\. A retrievedj=0j\{=\}0control separately measures selection without depth reuse\.

## 3Motivation

CoMem follows from one empirical property—transformer depth is not used uniformly—and three consequences for memory design\. We keep only the evidence needed to motivate them here; controlled ablations are in §[5](https://arxiv.org/html/2607.28263#S5)and Appendix[A\.1](https://arxiv.org/html/2607.28263#A1.SS1)\.

##### \(I1\) Cache an intermediate residual tensor\.

Layer\-wise probes place semantic accessibility near mid\-depth, while causal truncation shows that a shallower prefix retains substantial downstream signal\. Upper states are more query\-conditioned\(Ladet al\.,[2024](https://arxiv.org/html/2607.28263#bib.bib25)\), so a cached query\-blind state should not be treated as final\. CoMem therefore stores the complete per\-token residualhjh\_\{j\}and recomputes\[j:L\]\[j\{:\}L\]with the query present\. On Qwen3\-8B,hjh\_\{j\}uses1/181/18the bytes of full bf16 KV per stored token\.

##### \(I2\) Treat depth as a quality–cost knob\.

Recall remains high for shallowjjand then falls as the cached state crosses a zero\-shot readable boundary \(Figure[2](https://arxiv.org/html/2607.28263#S3.F2)a\)\. This boundary is distinct from the semantic*content*peak and moves deeper with model scale \(Figure[2](https://arxiv.org/html/2607.28263#S3.F2)b\)\. We exposejjas a deployment knob and use self\-distillation to move the operating point deeper; the flagship usesj=12/36j\{=\}12/36\. Exactj=0j\{=\}0equivalence and controlled depth/adaptation results appear in §[5](https://arxiv.org/html/2607.28263#S5)\.

![Refer to caption](https://arxiv.org/html/2607.28263v1/x2.png)Figure 2:The depth division of labor sets a bounded, tunable cache depth\.\(a\) On Qwen3\-8B, zero\-shot needle recall stays perfect while the splitjjis shallow and then collapses once the cache enters the query\-conditioned band; a light self\-distilled adapter pushes the readable depth deeper\. Panel \(a\) is an independent trend sweep; Table[10](https://arxiv.org/html/2607.28263#A1.T10)gives the controlled fixed\-jjcohort\. \(b\) Across scale, the semantic*content*depth is near scale\-invariant \(≈\\approx0\.45​L0\.45L\) while the zero\-shot*readable*depth deepens with model size; the shaded gap is the adaptation headroom suggested by the probe, and it vanishes by 32B\. Full tables are in Appendix[A\.1](https://arxiv.org/html/2607.28263#A1.SS1)\.
##### \(I3\) Read a bounded relevant pack\.

With the gold chunk present, an oracle selector reaches100%100\\%on the reported RULER needle sweep, whereas recency and reader\-attention selection fail as length grows \(Figure[3](https://arxiv.org/html/2607.28263#S3.F3)\)\. The observed bottleneck on this diagnostic is therefore selection, not cached\-state fidelity\. CoMem uses bounded relevance retrieval and full cross\-chunk upper\-layer attention; §[5](https://arxiv.org/html/2607.28263#S5)tests retrieval breadth and chunk interaction directly\.

![Refer to caption](https://arxiv.org/html/2607.28263v1/x3.png)Figure 3:Selection, not fidelity, is the observed long\-range bottleneck\.On the chat\-template\-free RULER single\-needle sweep, Oracle and BM25 remain at 100, while recency and reader attention fall at 32k\. Full selector results are in Appendix Table[12](https://arxiv.org/html/2607.28263#A1.T12)\.

## 4Methodology

CoMem partitions a decoder along depth \(Figure[4](https://arxiv.org/html/2607.28263#S4.F4)\): Write caches one intermediate residual tensor per chunk; Select retrieves a bounded subset; Read recomputes the upper layers with the query present\. The resumed computation is related to KV\-Direct\(Qasimet al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib24)\), but CoMem combines it with shallow writes, a tunable split, and bounded external retrieval\.

![Refer to caption](https://arxiv.org/html/2607.28263v1/x4.png)Figure 4:CoMem pipeline\.Writeencodes each 512\-token chunk through\[0:j\]\[0\{:\}j\]and cacheshjh\_\{j\}\.Selectretrieves top\-kkrelevant chunks\.Readpacks their hiddens with a sink and the query, then recomputes\[j:L\]\[j\{:\}L\]with full cross\-chunk attention\. The flagship reads approximately 6\.5k tokens regardless of stored\-context length\.##### Notation and storage\.

Let the decoder haveLLlayers, widthdd, per\-layer KV widthdkvd\_\{\\mathrm\{kv\}\}, splitjj, and chunk sizecc\. CoMem storeshj∈ℝc×dh\_\{j\}\\in\\mathbb\{R\}^\{c\\times d\}\. With a common dtype, its per\-token storage relative to full KV is

\|hj\|\|KV\|=d2​L​dkv=nq2​L​nkv,\\frac\{\\lvert h\_\{j\}\\rvert\}\{\\lvert\\mathrm\{KV\}\\rvert\}=\\frac\{d\}\{2Ld\_\{\\mathrm\{kv\}\}\}=\\frac\{n\_\{\\mathrm\{q\}\}\}\{2Ln\_\{\\mathrm\{kv\}\}\},\(1\)wherenqn\_\{\\mathrm\{q\}\}andnkvn\_\{\\mathrm\{kv\}\}are the query\- and KV\-head counts\. Qwen3\-8B usesd=4096d\{=\}4096,L=36L\{=\}36,nq=32n\_\{\\mathrm\{q\}\}\{=\}32, andnkv=8n\_\{\\mathrm\{kv\}\}\{=\}8, yielding1/181/18\(8 KB vs\. 144 KB per token in bf16\)\. At read time we assemble a*pack*whose length is fixed bykkandcc,

\|pack\|=sink\+k⋅c\+query≈6\.7​k,\\lvert\\text\{pack\}\\rvert=\\text\{sink\}\+k\\\!\\cdot\\\!c\+\\text\{query\}\\approx 6\.7\\text\{k\},\(2\)fork=12k\{=\}12,c=512c\{=\}512, and—crucially—*independent of the context length*\.

Algorithm 1CoMem: Write and Read0:backbone layers

\[0:L\]\[0\{:\}L\], split

jj, chunk size

cc, top\-

kk
1:Write\(streaming, once per context\):

2:foreach

cc\-token chunk

xxin the contextdo

3:

hj←layers\[0:j\]\(x\)h\_\{j\}\\leftarrow\\mathrm\{layers\}\[0\{:\}j\]\(x\)\{chunk\-local RoPE\}

4:store

hjh\_\{j\}keyed by the chunk text

5:endfor

6:Read\(per query

qq\):

7:

S←Lexical\-top\-​k​\(q\)S\\leftarrow\\textsc\{Lexical\-top\-\}k\(q\)\{bounded top\-12 flagship retrieval\}

8:

pack←\[sink;\{hj\}s∈S;hj​\(q\)\]\\mathrm\{pack\}\\leftarrow\[\\,\\mathrm\{sink\};\\ \\\{h\_\{j\}\\\}\_\{s\\in S\};\\ h\_\{j\}\(q\)\\,\]\{fresh contiguous RoPE\}

9:

logits←layers\[j:L\]\(pack\)\\mathrm\{logits\}\\leftarrow\\mathrm\{layers\}\[j\{:\}L\]\(\\mathrm\{pack\}\)\{full cross\-pack attention\}

10:returnnext\-token

logits\\mathrm\{logits\}

### 4\.1What to cache: the Write pass

The document is streamed in disjointcc\-token chunks\. Each chunk restarts rotary positions\(Suet al\.,[2021](https://arxiv.org/html/2607.28263#bib.bib35)\)at zero, runs through\[0:j\]\[0\{:\}j\], and writeshjh\_\{j\}to an external store keyed by its text\. Chunk\-local positions make cached states reusable regardless of their original document offset\. The Write cost is linear in context length and executes onlyjjlayers\. The hidden store and BM25 index also remain linear; only the model\-side online Read is bounded\.

### 4\.2How deep: the splitjjand self\-distillation

##### Thejjknob\.

Split depth controls prepaid Write work versus per\-query Read work\. Atj=0j\{=\}0, CoMem is a selective full forward and matches the stock forward withinmax⁡\|Δ​logit\|<10−4\\max\|\\Delta\\mathrm\{logit\}\|<10^\{\-4\}on the same pack; atj=Lj\{=\}L, no upper layer is replayed\. Intermediatejjtrades cheaper Read for query\-blind readout loss\. Self\-distillation enables the flagship operating pointj=12/36j\{=\}12/36\. KV\-Direct is the separate no\-retrieval full\-context arm\.

##### Self\-distillation\.

Following knowledge distillation and low\-rank adaptation\(Hintonet al\.,[2015](https://arxiv.org/html/2607.28263#bib.bib40); Huet al\.,[2022](https://arxiv.org/html/2607.28263#bib.bib37)\), we train a student atj=12j\{=\}12to match a frozenj=0j\{=\}0teacher on the teacher’s top\-64 logit support\. Letppandqqbe their renormalized distributions on that support\. The actual objective is the bidirectional KL

ℒdistill=0\.6​KL​\(p∥q\)\+0\.4​KL​\(q∥p\),\\mathcal\{L\}\_\{\\mathrm\{distill\}\}=0\.6\\,\\mathrm\{KL\}\(p\\\|q\)\+0\.4\\,\\mathrm\{KL\}\(q\\\|p\),\(3\)with no auxiliary cross\-entropy term\. The canonical adapter trains rank\-32 LoRA \(α=64\\alpha\{=\}64, zero dropout\) on all attention and MLP projections in layers 12–35 of a frozen Qwen3\-8B\. Training uses 4,000 steps of 8\-GPU bf16 DDP \(effective batch 8\), AdamW with peak learning rate8×10−58\{\\times\}10^\{\-5\}, 100 warmup steps and cosine decay, on 2,048\-token windows from the PG\-19 training split\(Raeet al\.,[2019](https://arxiv.org/html/2607.28263#bib.bib13)\)\. No synthetic long\-context examples, task supervision, retrieval, or CE loss is used\. This pushes the usable cache depth deeper and transfers without benchmark\-specific supervision across the evaluated tasks \(§[5](https://arxiv.org/html/2607.28263#S5)\)\.

The backbone remains frozen\. Thej=12j\{=\}12flagship uses this LoRA, while a shallower adapter\-free arm isolates the inference architecture\. Designing compressible cache points during pretraining is complementary and outside our scope\.

### 4\.3How to read: retrieval and recompute

##### The read pack\.

A selector returns top\-kkchunk indices\. Their hiddens are ordered by document position and packed as\[sink; selected​hj​; query​hj​\]\\texttt\{\[sink; selected \}h\_\{j\}\\texttt\{; query \}h\_\{j\}\\texttt\{\]\}with fresh, contiguous RoPE\. Layers\[j:L\]\[j\{:\}L\]then run with full causal attention, allowing later chunks and the query to integrate earlier evidence\. For fixedkkandcc, model\-side Read FLOPs and KV working memory are independent of stored\-context length; Table[6](https://arxiv.org/html/2607.28263#S5.T6)tests the full\-attention choice\.

For generation, CoMem prefills the resumed upper\-band KV once and maintains a query\-local lower\-band cache; it does not rerun the full pack per token\. Decode work is therefore independent of stored\-context length for a fixed pack and generated prefix\. Table[3](https://arxiv.org/html/2607.28263#S5.T3)verifies this path and reports timing\.

##### Bounded lexical retrieval\.

BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.28263#bib.bib34)\)returns a bounded top\-kkset\. The flagship usesk=12k\{=\}12, hop width four, and three automatic frontier\-expansion rounds with deduplication\. Oracle, recency, reader\-attention, and wider retrieval are analysis\-only controls reported in §[5](https://arxiv.org/html/2607.28263#S5)and Appendix[A\.1](https://arxiv.org/html/2607.28263#A1.SS1)\.

## 5Experiments

##### Models and unified protocol\.

Unless noted, the backbone is theQwen3\-8B base LM\(Yang and others,[2025](https://arxiv.org/html/2607.28263#bib.bib31)\)\(36 layers\), continued\-trained without SFT or RL\. All methods are evaluated under a unified plain\-text prompting protocol, with chat templating disabled \(chat\_template=False\)\. The backbone remains frozen; the flagship trains only our self\-distilled rank\-32 LoRA at splitj=12j\{=\}12and uses chunk size 512, a BOS sink, anditer\_bm25\. All five benchmarks fix top\-1212, hop width four, androunds=0, which automatically executes three retrieval rounds; the observed read is approximately 6\.5k tokens\. The native window is 40,960 tokens; 131,072 is an inactive YaRN\(Penget al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib36)\)extrapolation ceiling\.

##### Baselines and position range\.

KV\-Direct\(Qasimet al\.,[2026](https://arxiv.org/html/2607.28263#bib.bib24)\), the HCache\-style reproduction\(Gaoet al\.,[2025](https://arxiv.org/html/2607.28263#bib.bib1)\), StreamingLLM\(Xiao and others,[2024b](https://arxiv.org/html/2607.28263#bib.bib6)\), and InfLLM\(Xiao and others,[2024a](https://arxiv.org/html/2607.28263#bib.bib5)\)use Qwen3\-8B\. KV\-Direct runs with native RoPE and no YaRN; its results above 40,960 tokens are therefore an*unextended*full\-context stress reference, not a substitute for a length\-extended model\. We claim continued usability where this unextended reference fails, not superiority to YaRN\-enabled full context\. For MemoryLLM\(Wang and others,[2024](https://arxiv.org/html/2607.28263#bib.bib22)\), we use the official Llama\-3\-8B\-chat backbone\(Grattafiori and others,[2024](https://arxiv.org/html/2607.28263#bib.bib32)\)checkpoint released with its paper\. It is an out\-of\-backbone reference\. Because this chat\-tuned checkpoint is evaluated without its native chat template, the resulting invocation is out of distribution and is interpreted only as a diagnostic reference\.

##### Evaluation & metrics\.

Synthetic RULER\(Hsieh and others,[2024](https://arxiv.org/html/2607.28263#bib.bib9)\)and BABILong\(Kuratov and others,[2024](https://arxiv.org/html/2607.28263#bib.bib10)\)tables contain 100 examples per task–length cell\. RULER uses officialstring\_match\_all; BABILong usescompare\_answerswithTASK\_LABELSand neverre\.search\. LongBench\(Bai and others,[2023](https://arxiv.org/html/2607.28263#bib.bib12)\)and LoCoMo\(Maharana and others,[2024](https://arxiv.org/html/2607.28263#bib.bib11)\)userun\_scoring\. LongEval follows the LongChat lines\-retrieval protocol\(Liet al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib14)\)\. LoCoMo reports deterministic token F1 and substring accuracy as lexical diagnostics, with semantic Judge as the primary metric: GPT\-4o scores categories 1–4, while category 5 uses the official local abstention rule\. Complete RULER grids passed the eight\-shard completeness check, contained no empty predictions, and matched scores recomputed from disk\. Appendix[B](https://arxiv.org/html/2607.28263#A2)gives aggregation and uncertainty details\.

##### Hardware\.

Headline Qwen3\-8B timing uses one 96 GB NVIDIA H20 \(median of three runs, chunk size 512\)\. At 128k, full\-context prefill peaks at 89\.36 GB and CoMem at 18\.26 GB\. A separate same\-H20 split\-depth sweep holds the retrieved pack fixed and is reported only within that cohort\. Large\-MoE details are in Appendix[A\.5](https://arxiv.org/html/2607.28263#A1.SS5)\.

Table 1:Cross\-benchmark comparison under the unified plain\-text evaluation protocol, with model\-specific chat templating disabled\. Avg\. is the unweighted mean of the five displayed benchmark metrics, each reported on a 0–100 scale\. Full\-context Qwen3 results beyond the native 40,960\-token window use native RoPE without YaRN and are therefore unextended stress references rather than length\-extended upper bounds\. The two CoMem rows separate the distilled and frozen configurations\.†MemoryLLM uses an out\-of\-backbone Llama\-3\-8B\-chat checkpoint and is included only as a diagnostic reference\.
### 5\.1Core benchmark overview

Table[1](https://arxiv.org/html/2607.28263#S5.T1)compares five audited benchmarks under the unified evaluation protocol described above\. CoMem scores 97\.05 on RULER and 69\.0 on LongEval versus 78\.80/65\.2 for the unextended KV\-Direct reference, while matching its LongBench macro F1 \(12\.15 vs\. 12\.17\)\. Frozenj=9j=9is weak on multi\-fact retrieval, and within\-window BABILong exposes a qa1/qa2 compression tax, whereas the distilled model leads qa5\. The advantage is therefore a bounded read, not uniform accuracy dominance; fidelity depends on selection, depth, and adaptation\. Appendix[A\.2](https://arxiv.org/html/2607.28263#A1.SS2)gives full grids\.

### 5\.2Equal\-budget comparison

At the same∼\\sim6\.5k\-token budget, CoMem stays near 100 on single needles and above 95 on variable tracking through 128k, while StreamingLLM falls 85→\\to2 and 41\.8→\\to0\.8 \(Appendix Table[13](https://arxiv.org/html/2607.28263#A1.T13)\)\. With read length approximately fixed, the gap shows that relevant rather than merely recent chunks are essential\.

### 5\.3Long\-dialogue memory

Table 2:LoCoMo comparison on all 1,986 chat\-template\-free examples\. Judge1:4is GPT\-4o accuracy on the 1,540 answerable items; the full Judge column folds in local abstention scoring for 446 adversarial items\. F1 and substring accuracy are deterministic lexical diagnostics; semantic Judge is the primary metric\. All seven rows use the same scoring protocol\. The distilled and frozen rows show the two deployment points; the same\-jjadapter effect is isolated in Table[8](https://arxiv.org/html/2607.28263#A1.T8)\. On paired answerable items, distilled CoMem exceeds KV\-Direct by4\.814\.81points \(item\-bootstrap 95% CI\[2\.34,7\.27\]\[2\.34,7\.27\]\); a 10\-conversation cluster bootstrap also excludes zero, and an independent DeepSeek\-V3 audit preserves the ordering\.†For MemoryLLM, we use the official chat\-tuned Llama\-3 checkpoint released with its paper; it is therefore an out\-of\-backbone reference\.CoMem scores 38\.27 versus 34\.59 for KV\-Direct on all 1,986 LoCoMo questions\. The advantage is not an artifact of one scorer or treating correlated questions as independent: on 1,540 paired answerable items, the\+4\.81\+4\.81gap has item\-bootstrap 95% CI\[2\.34,7\.27\]\[2\.34,7\.27\]; a 10\-conversation cluster bootstrap remains positive \(8/10 conversations favor CoMem\); and a stratified 200\-item DeepSeek\-V3 audit preserves the ordering \(κ=0\.626\\kappa=0\.626with GPT\-4o\)\. Retrievedj=0j=0reaches 41\.59, showing that retrieval contributes independently of depth reuse, while the same\-jjadapter control below measures how much fidelity is recovered\. Appendix[B](https://arxiv.org/html/2607.28263#A2)details both uncertainty analyses and judge prompts\.

### 5\.4End\-to\-end efficiency

\(a\) Full\-write\-inclusive prefill

\(b\) Resumed\-band KV\-cache decode

Table 3:Qwen3\-8B end\-to\-end efficiency \(j=12j\{=\}12, bf16, SDPA, H20\-class platforms\)\.\(a\)Adapter\-free full\-write\-inclusive prefill: at 128k, CoMem is7\.83×7\.83\\timesfaster and uses 18\.26 GB versus 89\.36 GB\.\(b\)Resumed\-band cached decode stays within 11% of Dense through 64k and is1\.07×1\.07\\timesfaster at 128k; the correctness gate passes\.Table 4:Same\-platform LoRA overhead at 128k\. LoRA adds 0\.25 GB peak memory and about 18% prefill time while preserving the fixed\-read advantage\. This is a separate paired\-platform control; its ratios are not mixed with Table[3](https://arxiv.org/html/2607.28263#S5.T3)\.Tables[3](https://arxiv.org/html/2607.28263#S5.T3)and[4](https://arxiv.org/html/2607.28263#S5.T4)separate three costs that are easily conflated\. First, full\-write\-inclusive prefill becomes favorable after 8k and reaches7\.83×7\.83\\timesat 128k while peak memory grows only from 17\.33 to 18\.26 GB; the full\-context reference reaches 89\.36 GB\. Second, deployed resumed\-band decode does not rerun the pack: it stays within 11% of Dense through 64k and becomes1\.07×1\.07\\timesfaster at 128k, with the correctness gate passing\. Third, the separate single\-column LoRA control shows that adaptation adds 0\.25 GB and retains a2\.74×2\.74\\timesprefill speedup\. A fixed\-pack depth sweep finds thatj=12j=12lowers Read by 29% relative toj=0j=0; this diagnostic isolates upper\-layer replay and is not mixed with cached decode\. Appendix[A\.4](https://arxiv.org/html/2607.28263#A1.SS4)gives the measurement protocol\.

### 5\.5What drives the result?

##### Retrieval versus depth reuse\.

The controlled depth sweep \(Appendix Table[8](https://arxiv.org/html/2607.28263#A1.T8)\) holds the retrieved pack fixed\. Thej=0j=0row performs full\-depth recomputation on it and therefore isolates selection from depth reuse\. It reaches 41\.59 on LoCoMo, compared with 34\.59 for full\-context KV\-Direct, showing that bounded selection can improve natural dialogue memory even before layer reuse\. Deeper frozen splits reduce quality \(32\.78/29\.15/24\.52 atj=6/9/12j=6/9/12\) as query\-blind states become harder to read, but a same\-pack 128k sweep reduces Read from 1\.01 to 0\.72 s\. Depth is therefore a quality–query\-cost knob rather than a free accuracy gain\.

##### Self\-distillation repairs readout fidelity\.

At the samej=12j=12, the PG19\-only LoRA restores 13\.75 LoCoMo points and 22\.2/9\.0/8\.4 BABILong qa1/qa2/qa5 points\. The controlled RULER cohort similarly moves single\-needle recall from 35/5/17 to 100/100/100 and multikey recall from 4/2/1 to 91/95/92 over 8k/16k/32k \(Appendix Table[10](https://arxiv.org/html/2607.28263#A1.T10)\)\. This supports adaptation of the split interface rather than benchmark\-specific training: PG19 contains no retrieval or task supervision\.

##### Selection breadth and chunk interaction\.

Table 5:Retrieval breadth on RULER variable tracking \(n=100n\{=\}100\) without a chat template\. The first row is the flagship automatic three\-round retrieval; the second widens the read to roughly 17k\. They are distinct configurations and not an equal\-budget causal comparison\. The expansion point estimate is higher at 8k and lower from 16k onward, consistent with the larger read introducing distractor variable chains\. More retrieval budget is not uniformly beneficial\.Table 6:Full causal cross\-chunk attention versus block\-diagonal reuse without a chat template \(8k/16k\)\. In both arms the final query attends to every retrieved chunk; the block\-diagonal arm prevents cached chunks from attending across chunk boundaries\. RULER usesn=50n\{=\}50and BABILongn=100n\{=\}100, with the same top\-12 iterative retrieval\. Joint recomputation matters most for multi\-fact disambiguation, while a single needle needs little interaction\.Oracle retrieval keeps the reported needle cells at 100, whereas recency and reader\-attention selectors degrade with length \(Appendix Table[12](https://arxiv.org/html/2607.28263#A1.T12)\)\. More retrieval is not automatically better: the∼\\sim17k four\-hop variant is worse than the focused∼\\sim6\.5k flagship from 16k onward on variable tracking \(Table[5](https://arxiv.org/html/2607.28263#S5.T5)\), consistent with additional distractor chains\. Once selected, chunks must interact: replacing full cross\-chunk attention by block\-diagonal reuse costs 36/62 points on 8k/16k multikey retrieval and 19/7 points on BABILong qa2 \(Table[6](https://arxiv.org/html/2607.28263#S5.T6)\)\. Together these controls justify bounded relevance selection, a tunable split, full cross\-chunk recomputation, and lightweight interface adaptation as distinct parts of CoMem\.

### 5\.6Additional tasks and generality

Appendix[A\.2](https://arxiv.org/html/2607.28263#A1.SS2)gives the remaining task details\. Across Qwen3 0\.6B–32B, the probed content depth stays near0\.45​L0\.45Lwhile the zero\-shot readable boundary moves deeper with scale; this is a depth trend, not a claim of complete task transfer at every size\. CoMem is also viable on two sparse MoE backbones\. On Hy3, it keeps a≈\\approx4\.3–4\.6k\-token read through 256k and reaches 100/98 single/multikey recall \(Appendix[A\.5](https://arxiv.org/html/2607.28263#A1.SS5)\)\.

## 6Conclusion

CoMem caches mid\-layer chunk states and recomputes query\-conditioned upper layers over a bounded, relevance\-selected pack\. Under the unified evaluation protocol, it reaches 97\.05 on RULER, 69\.0 on LongEval, 12\.15 on LongBench, and 55\.6/27\.0/68\.7 on BABILong\. Its LoCoMo score is 38\.27, compared with 34\.59 for KV\-Direct; the gap remains positive under a conversation\-level cluster bootstrap and is preserved by an independent judge audit\. Same\-pack sweeps expose the central trade\-off: deeper splits precompute more work and reduce query\-time cost, but sacrifice frozen readout fidelity, while thej=12j=12LoRA recovers 13\.75 LoCoMo points without updating the backbone\. CoMem therefore offers a quality–memory–compute trade\-off rather than suggesting that depth cutting alone improves accuracy\. Across Qwen3 model scales and two sparse MoE backbones, the results support depth as a complementary reuse axis for long\-context memory\.

## 7Limitations

CoMem is a bounded\-read efficiency method, not an accuracy state of the art for every in\-window task\. BABILong qa2 remains weak \(27\.0\), and full context is a strong reference whenever it fits\. Qwen3\-8B is not position\-extended in our experiments: results above its native 40,960\-token window are unextended stress references and do not establish superiority over a YaRN\-enabled full\-context model\. Retrieval breadth is task\-dependent; retrieval latency on natural multi\-hop workloads remains unmeasured\. Full cross\-chunk attention improves multi\-fact accuracy but increases read compute\.

Our evaluation is English\-only and emphasizes retrieval\-style QA rather than long\-form generation, code, multilingual, or multimodal memory\. We use greedy decoding, a 512\-token flagship chunk size, and fixed top\-12 BM25 retrieval; learned/dense retrievers, dynamic index updates, deletions, and continually changing memory streams are outside the present scope\. Lexical retrieval may be less suitable when relevant passages share little surface form with the query\.

Efficiency measurements use bf16/SDPA on H20\-class hardware and a single\-query harness\. Kernel choice, batching, concurrent serving, storage I/O, and a networked external store can change absolute latency\. Write is reusable and is most attractive when a context serves multiple queries; for one short query, preprocessing may not amortize\. We do not measure index\-build energy or total persistent\-store capacity\.

LoCoMo contains only 10 conversation clusters\. We therefore supplement the item\-level interval with conversation\-cluster bootstrap and a stratified DeepSeek\-V3 audit, but neither replaces evaluation on more conversations\. The GPT\-4o endpoint also lacks a dated snapshot\. Synthetic cells use 100 examples, and not every ablation has multiple seeds or a paired test; chunk and attention differences are consequently reported as point estimates\.

## 8Ethical Considerations

CoMem changes the cost and storage organization of an underlying language model; it does not add a safety mechanism\. It therefore inherits risks such as hallucination, biased generation, unsafe completion, and unintended disclosure of sensitive text selected from a memory store\. Lower\-cost access to long histories could also scale benign applications and deliberate misuse alike\. Deployments should enforce access control on the persistent store, filter or redact sensitive sources before Write, respect deletion requests, restrict retrieval to authorized material, and retain application\-appropriate output monitoring and human review\. Although bounded reads reduce accelerator memory and per\-query computation, training and large\-scale evaluation still consume energy; we report the measured hardware and final training budget in Appendix[A\.3](https://arxiv.org/html/2607.28263#A1.SS3)\.

We collect no new human\-subject data and recruit no annotators\. Our experiments use previously released model, corpus, and benchmark artifacts; LoCoMo’s released conversations are generated by paired language\-model agents rather than newly collected private conversations\. The submitted software archive contains code only and excludes model weights, benchmark text, predictions, credentials, and API responses\.

## References

- LongBench: a bilingual, multitask benchmark for long context understanding\.arXiv preprint arXiv:2308\.14508\.Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px3.p1.1)\.
- Z\. Caiet al\.\(2024\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Gao, Y\. Chen, and J\. Shu \(2025\)Fast state restoration in LLM serving with HCache\.InProceedings of the Twentieth European Conference on Computer Systems \(EuroSys\),Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Gelberg, Y\. Eitan, M\. Bronstein, Y\. Gal, and H\. Maron \(2026\)Training transformers for KV cache compressibility\.arXiv preprint arXiv:2605\.05971\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- A\. Grattafioriet al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:2407\.21783Cited by:[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p3.2),[§4\.2](https://arxiv.org/html/2607.28263#S4.SS2.SSS0.Px2.p1.4)\.
- C\. Hsiehet al\.\(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p3.2),[§4\.2](https://arxiv.org/html/2607.28263#S4.SS2.SSS0.Px2.p1.4)\.
- C\. Jinet al\.\(2024\)RAGCache: efficient knowledge caching for retrieval\-augmented generation\.arXiv preprint arXiv:2404\.12457\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- Y\. Kuratovet al\.\(2024\)BABILong: testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.InNeurIPS Datasets and Benchmarks,Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px3.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p1.2)\.
- V\. Lad, W\. Gurnee, and M\. Tegmark \(2024\)The remarkable robustness of LLMs: stages of inference?\.arXiv preprint arXiv:2406\.19384\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.28263#S3.SS0.SSS0.Px1.p1.4)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Kuettler, M\. Lewis, W\. Yih, T\. Rocktaschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- D\. Li, R\. Shao, A\. Xie, Y\. Sheng, L\. Zheng, J\. E\. Gonzalez, I\. Stoica, X\. Ma, and H\. Zhang \(2023\)How long can open\-source LLMs truly promise on context length?\.Note:LMSYS Org BlogExternal Links:[Link](https://lmsys.org/blog/2023-06-29-longchat)Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px3.p1.1)\.
- Y\. Liet al\.\(2024\)SnapKV: LLM knows what you are looking for before generation\.arXiv preprint arXiv:2404\.14469\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Lin, J\. Wang, O\. Kondrateva, Y\. Shi, B\. Li, and G\. L\. Zhang \(2026\)CompressKV: semantic\-retrieval\-guided KV\-cache compression for resource\-efficient long\-context LLM inference\.arXiv preprint arXiv:2606\.24467\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Liu, J\. Liu, Z\. Pan, Y\. He, G\. Haffari, and B\. Zhuang \(2024a\)MiniCache: KV cache compression in depth dimension for large language models\.arXiv preprint arXiv:2405\.14366\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang, C\. Chen, F\. Yang, Y\. Yang, and L\. Qiu \(2024b\)RetrievalAttention: accelerating long\-context LLM inference via vector retrieval\.arXiv preprint arXiv:2409\.10516\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- A\. Maharanaet al\.\(2024\)Evaluating very long\-term conversational memory of LLM agents\.arXiv preprint arXiv:2402\.17753\.Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px3.p1.1)\.
- A\. Mohtashami and M\. Jaggi \(2023\)Landmark attention: random\-access infinite context length for transformers\.arXiv preprint arXiv:2305\.16300\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- B\. Peng, J\. Quesnelle, H\. Fan, and E\. Shippole \(2023\)YaRN: efficient context window extension of large language models\.arXiv preprint arXiv:2309\.00071\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p1.2),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px1.p1.2)\.
- K\. U\. Qasim, J\. Zhang, M\. K\. Shaheen, R\. Alharith, and H\. Zhang \(2026\)The residual stream is all you need: on the redundancy of the KV cache in transformer inference\.arXiv preprint arXiv:2603\.19664\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.28263#S4.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Qiu, X\. Qi, H\. Wang, X\. Juan, Y\. Wang, Z\. Zhao, J\. Geng, J\. Guo, P\. Li, J\. Shi,et al\.\(2025a\)Alita\-g: self\-evolving generative agent for agent generation\.arXiv preprint arXiv:2510\.23601\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1)\.
- J\. Qiu, X\. Qi, T\. Zhang, X\. Juan, J\. Guo, Y\. Lu, Y\. Wang, Z\. Yao, Q\. Ren, X\. Jiang,et al\.\(2025b\)Alita: generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self\-evolution\.arXiv preprint arXiv:2505\.20286\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. Lillicrap \(2019\)Compressive transformers for long\-range sequence modelling\.arXiv preprint arXiv:1911\.05507\.Cited by:[§A\.3](https://arxiv.org/html/2607.28263#A1.SS3.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.28263#S4.SS2.SSS0.Px2.p1.6)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2),[§4\.3](https://arxiv.org/html/2607.28263#S4.SS3.SSS0.Px2.p1.2)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2021\)RoFormer: enhanced transformer with rotary position embedding\.arXiv preprint arXiv:2104\.09864\.Cited by:[§4\.1](https://arxiv.org/html/2607.28263#S4.SS1.p1.4)\.
- Y\. Sunet al\.\(2024\)You only cache once: decoder\-decoder architectures for language models\.arXiv preprint arXiv:2405\.05254\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context LLM inference\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- S\. Tworkowskiet al\.\(2023\)Focused transformer: contrastive training for context scaling\.arXiv preprint arXiv:2307\.03170\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5998–6008\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p1.2)\.
- Q\. Wanget al\.\(2023\)Recursively summarizing enables long\-term dialogue memory in large language models\.arXiv preprint arXiv:2308\.15022\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- W\. Wang, L\. Dong, H\. Cheng, X\. Liu, X\. Yan, J\. Gao, and F\. Wei \(2023\)Augmenting language models with long\-term memory\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- Y\. Wanget al\.\(2024\)MemoryLLM: towards self\-updatable large language models\.arXiv preprint arXiv:2402\.04624\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- H\. Wu and K\. Tu \(2024\)Layer\-condensed KV cache for efficient inference of large language models\.arXiv preprint arXiv:2405\.10637\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Xiaoet al\.\(2024a\)InfLLM: training\-free long\-context extrapolation with an efficient context memory\.arXiv preprint arXiv:2402\.04617\.Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- G\. Xiaoet al\.\(2024b\)Efficient streaming language models with attention sinks\.InICLR,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px2.p1.1)\.
- A\. Yanget al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:2505\.09388Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p1.2),[§5](https://arxiv.org/html/2607.28263#S5.SS0.SSS0.Px1.p1.2)\.
- J\. Yaoet al\.\(2024\)CacheBlend: fast large language model serving for RAG with cached knowledge fusion\.arXiv preprint arXiv:2405\.16444\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px3.p1.2)\.
- H\. Yen, T\. Gao, and D\. Chen \(2024\)Long\-context language modeling with parallel context encoding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- P\. Zhanget al\.\(2024\)Long context compression with activation beacon\.arXiv preprint arXiv:2401\.03462\.Cited by:[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px2.p1.2)\.
- Z\. Zhanget al\.\(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2607.28263#S1.p2.1),[§2](https://arxiv.org/html/2607.28263#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Results and Ablations

This appendix contains the detailed task tables, mechanism ablations, and cross\-scale generality results summarized in the eight\-page main paper\. Appendix[B](https://arxiv.org/html/2607.28263#A2)separately documents aggregation, uncertainty, and the LoCoMo judge protocol\.

### A\.1Mechanism ablations

##### Split depth and self\-distillation\.

Table[7](https://arxiv.org/html/2607.28263#A1.T7)reports semantic*content*and adapter\-off*readable*depth across Qwen3 scales\. Table[8](https://arxiv.org/html/2607.28263#A1.T8)gives the frozenj=0/6/9/12j=0/6/9/12quality sweep and same\-jjdistilled control; Table[9](https://arxiv.org/html/2607.28263#A1.T9)pairs the same depths with a fixed\-pack, same\-H20 cost sweep\. Table[10](https://arxiv.org/html/2607.28263#A1.T10)provides a separate same\-jjRULER control\. Finally, Table[11](https://arxiv.org/html/2607.28263#A1.T11)applies the same adapter to a retrieval\-free HCache path, showing that it is a portable mid\-layer readout repair rather than a retrieval\-specific component\.

Table 7:Depth partition: content vs\. readable depth\.The*content\-jj*\(the knee98 depth from a truncation\-downstream linear probe\) sits at≈\\approx0\.45​L0\.45Land is*near scale\-invariant*over the evaluated 0\.6B–32B range\. The zero\-shot*readout\-crash*depth \(where CoMem’s zero\-shot single recall drops below 50% on RULERniah\_single16k\) instead*deepens monotonically with scale*, from0\.09​L0\.09L\(0\.6B\) to\>0\.42​L\>0\.42L\(32B, no crash observed\)\. Their difference is thegapthe self\-distilled adapter is intended to close: it is largest for the smallest model \(0\.37LLat 0\.6B\) and vanishes by 32B, where zero\-shot readout already reaches the content peak\. This trend predicts greater adaptation headroom at small scale; it is not a measured cross\-scale adapter\-gain ranking\.Table 8:Controlled comparison of retrieval, split depth, and self\-distillation using Qwen3\-8B\. All configurations follow the same plain\-text prompting protocol, with model\-specific chat templating disabled \(chat\_template=False\), and retrieve the same top\-12 pack\. For the frozen configurations, readout fidelity decreases as the split moves deeper\. At the same splitj=12j\{=\}12, the rank\-32 LoRA improves the LoCoMo Judge score by 13\.75 points and the BABILong qa1, qa2, and qa5 means by 22\.2, 9\.0, and 8\.4 points, respectively\. Thej=0j\{=\}0configuration performs full\-depth recomputation over the retrieved pack, thereby isolating the effect of relevance\-based selection; it provides no lower\-layer reuse and thus no depth\-based reduction in query\-time computation\.\(a\) Online Read latency

\(b\) Step, write, and memory at 128k

Table 9:Same\-H20 split\-depth cost sweep with no LoRA \(median of three runs\)\. Rows share an identical top\-12 pack \(about 6,657 tokens\), chunk 512, and retrieval result\. Fromj=0j=0\(retrieve then recompute all layers\) toj=12j=12, 128k Read falls 29% and the uncached packed\-step diagnostic 28%; the deeper split instead prepays reusable offline Write\. The step diagnostic isolates upper\-layer cost and is not the deployed KV\-cache decode in Table[3](https://arxiv.org/html/2607.28263#S5.T3)\.Table 10:Separate controlled RULER self\-distillation ablation on Qwen3\-8B at the same splitj=12j\{=\}12\(n=100n\{=\}100per cell\)\. Each on/off pair uses the same task, length, selector, and cohort, so the difference isolates the adapter\. Table[8](https://arxiv.org/html/2607.28263#A1.T8)provides the broader same\-jjLoCoMo/BABILong control used in the main argument\.Table 11:Portable readout adaptation on a retrieval\-free HCache path\. The two configurations use the same compute node, code commit, evaluation harness, split, and plain\-text prompting protocol, with model\-specific chat templating disabled \(chat\_template=False\)\. The distilled LoRA is the only changed component and improves the LoCoMo Judge score by 17\.88 points\. Because this within\-harness control belongs to a different measurement cohort, its values should not be pooled with the canonical cross\-node HCache result\.
##### Selection and retrieval breadth\.

Table[12](https://arxiv.org/html/2607.28263#A1.T12)compares Oracle, BM25, recency, and reader attention\. Table[5](https://arxiv.org/html/2607.28263#S5.T5)in the main paper contrasts the focused∼\\sim6\.5k configuration with a∼\\sim17k four\-hop expansion; these rows have different read budgets and are not a controlled per\-hop causal estimate\.

Table 12:Single\-pass selector sweep on RULER \(n=100n\{=\}100\) without a chat template; each entry is the peak overk∈\{4,8,12,16,24\}k\\in\\\{4,8,12,16,24\\\}\. Oracle retrieval keeps both needle tasks at 100, showing that their long\-range gap is selection rather than readout loss\. The variable\-tracking rows belong to this specific selector\-sweep configuration and should not be conflated with the focused top\-12 flagship in Table[13](https://arxiv.org/html/2607.28263#A1.T13)\. Oracle is undefined for a multi\-chunk reference chain and is therefore not reported there\.Table 13:Equal\-budget, chat\-template\-free RULER comparison \(n=100n\{=\}100per cell\)\. CoMem uses chunk 512, top\-12, a BOS sink, and a measured read of about 6\.5k tokens; StreamingLLM retains the same nominal token budget by recency\. Relevance\-based retrieval remains stable while recency truncation degrades with source length\.
##### Read interaction and chunk size\.

Table[6](https://arxiv.org/html/2607.28263#S5.T6)in the main paper compares full cross\-chunk attention with block\-diagonal reuse\. Table[14](https://arxiv.org/html/2607.28263#A1.T14)reports the chunk\-size sweep; its entries are point estimates rather than a multi\-seed stability ranking\.

Table 14:Chunk\-size trade\-off: observed RULERniah\_multikeyaccuracy \(n=100n\{=\}100\) and same\-harness packed\-step cost without a chat template\. Cost is approximately linear in read length\. It is an uncached full\-pack diagnostic, not the deployed KV\-cache decode in Table[3](https://arxiv.org/html/2607.28263#S5.T3)\. Accuracy differences are point estimates rather than a multi\-seed stability claim\.

### A\.2Full chat\-template\-free benchmark comparisons

Tables[15](https://arxiv.org/html/2607.28263#A1.T15)–[18](https://arxiv.org/html/2607.28263#A1.T18)give the full audited baseline breakdowns for RULER, LongEval, BABILong, and LongBench\. Every row uses the unified no\-chat protocol; MemoryLLM remains an out\-of\-backbone Llama\-3 reference\. The all\-method LoCoMo comparison is Table[2](https://arxiv.org/html/2607.28263#S5.T2); judge details are in Appendix[B](https://arxiv.org/html/2607.28263#A2)\.

MethodTask8k16k32k64k128kCoMem \+ LoRAsingle1001009999100multikey9594979193var\-track96\.697\.698\.899\.095\.8CoMem frozen \(j=9j\{=\}9\)single9699999996multikey4444593048var\-track4538\.836\.431\.225\.8KV\-Directsingle1001001001000multikey10010098880var\-track10010099\.896\.20InfLLMsingle100100926157multikey10079654520var\-track10096\.890\.681\.879\.2StreamingLLMsingle853620102multikey833418134var\-track41\.82\.20\.60\.20\.8HCache\-stylesingle335300multikey84000var\-track1\.61\.00\.400MemoryLLM†single2940372121multikey2824251210var\-track01\.2000

Table 15:Full chat\-template\-free RULER comparison using officialstring\_match\_all\(n=100n\{=\}100per cell\)\. KV\-Direct places the full context in the read with native RoPE and no YaRN; its 64k/128k cells are unextended stress references rather than a length\-extended upper bound\. CoMem keeps a bounded retrieved pack\. The frozen arm remains strong on a single needle but degrades on multikey and tracking\. InfLLM remains competitive in\-window but degrades on long needle tasks, while CoMem stays above 91 throughout\.†MemoryLLM uses its released Llama\-3\-8B\-chat checkpoint\.Table 16:LongEval register\-content accuracy\. Unmarked rows use the unified chat\-template\-free protocol\. Means use the common 8k–128k support\. KV\-Direct uses native RoPE without YaRN, so its 64k/128k cells are unextended stress references and the five\-length mean is not a length\-extended upper bound\. CoMem’s separate six\-length headline is 72\.83 after including its 4k score of 92\. The frozen CoMem control used 48 generation tokens rather than 16; both budgets exceed the single\-number answer length\.†MemoryLLM uses a different chat\-tuned backbone\.Table 17:BABILong accuracy \(n=100n\{=\}100per task–length cell\), evaluated with the officialcompare\_answersscorer andTASK\_LABELS\. Within the native context window, full\-context KV\-Direct is substantially stronger on qa1 and qa2, exposing the compression tax, whereas distilled CoMem achieves the highest qa5 mean, ahead of InfLLM \(68\.7 vs\. 64\.9\)\. All methods are evaluated under the unified plain\-text prompting protocol, with model\-specific chat templating disabled \(chat\_template=False\)\.†MemoryLLM uses the official Llama\-3\-8B\-chat checkpoint rather than the Qwen3\-8B backbone\. Because it is evaluated without its native chat template, this setting lies outside the checkpoint’s intended prompting distribution; its results are included only as an out\-of\-backbone diagnostic reference\.Table 18:Official LongBench QA token F1 by dataset\. All methods are evaluated under the same plain\-text prompting protocol, with model\-specific chat templating disabled \(chat\_template=False\)\. CoMem with LoRA achieves a macro F1 of 12\.15, closely matching the 12\.17 obtained by full\-context KV\-Direct, suggesting that bounded compression introduces little additional loss relative to the full\-context reference under this protocol\. The frozen CoMem configuration reports adapter\-free performance atj=9j\{=\}9\. NQA denotes NarrativeQA, and MultiF\. denotes MultiFieldQA\-en\.†MemoryLLM uses the official Llama\-3\-8B\-chat checkpoint rather than the Qwen3\-8B backbone\. Because it is evaluated without its native chat template, this setting lies outside the checkpoint’s intended prompting distribution; its results are included only as an out\-of\-backbone diagnostic reference\.
### A\.3Reproducibility details

Tables[21](https://arxiv.org/html/2607.28263#A1.T21)–[21](https://arxiv.org/html/2607.28263#A1.T21)record the configuration, training, and evaluation settings used by the Qwen3\-8B flagship\. The archived configuration files and evaluation scripts accompany the code release\.

Table 19:Flagship CoMem inference configuration\. The retrieval and read configuration is frozen across all five benchmarks; only benchmark harness parameters differ\.Table 20:Self\-distillation and LoRA training configuration\. The adapter name’s “4k” denotes 4,000 optimization steps; the training sequence length is 2,048 tokens\.Table 21:Benchmark\-level evaluation settings\. Every main comparison useschat\_template=False\. MemoryLLM uses a different chat\-tuned backbone, so its no\-chat row is interpreted only as an out\-of\-backbone reference\.##### Artifacts, licenses, and data scope\.

The study evaluates English text only\. It uses Qwen3\-8B\-Base \(Apache\-2\.0\), PG\-19\(Raeet al\.,[2019](https://arxiv.org/html/2607.28263#bib.bib13)\), RULER\(Hsieh and others,[2024](https://arxiv.org/html/2607.28263#bib.bib9)\), BABILong\(Kuratov and others,[2024](https://arxiv.org/html/2607.28263#bib.bib10)\), LongBench\(Bai and others,[2023](https://arxiv.org/html/2607.28263#bib.bib12)\), LongEval/LongChat\(Liet al\.,[2023](https://arxiv.org/html/2607.28263#bib.bib14)\), and LoCoMo\(Maharana and others,[2024](https://arxiv.org/html/2607.28263#bib.bib11)\)\. RULER and LongChat code are Apache\-2\.0; the LongBench repository is MIT\-licensed, while its component datasets retain their upstream terms; BABILong code is Apache\-2\.0 and incorporates BSD\-licensed bAbI tasks; LoCoMo is CC BY\-NC 4\.0\. PG\-19 contains pre\-1919 Project Gutenberg books, for which the dataset page does not assert one uniform corpus license, so we do not redistribute its text\. We likewise redistribute no model weights, benchmark examples, predictions, or API responses\. The anonymous code archive contains source, documentation, pinned environment requirements, and third\-party notices; users obtain all external artifacts from their original providers and remain responsible for their terms\. Table[21](https://arxiv.org/html/2607.28263#A1.T21)reports the evaluated supports and sample counts; LoCoMo category denominators are in Appendix[B](https://arxiv.org/html/2607.28263#A2)\.

##### Software environment\.

The archived environment specifies Python 3\.10\+, PyTorch 2\.10, Transformers 5\.5\.4, PEFT 0\.10\+, Datasets 2\.14\+, bf16, and SDPA\. Evaluation uses the benchmark\-specific official scorers named in Table[21](https://arxiv.org/html/2607.28263#A1.T21); the archive records all non\-default flags and provides a CPU correctness gate\.

##### Compute accounting\.

The final adapter run used one node with 8 H20 GPUs for about 22 minutes, or approximately 2\.9 H20 GPU\-hours\. Tables[21](https://arxiv.org/html/2607.28263#A1.T21)and[3](https://arxiv.org/html/2607.28263#S5.T3)report model size, trainable parameters, hardware, and measured timing protocol\. Total GPU\-hours across preliminary probes, failed runs, baseline generation, and all ablations were not consistently logged; we report this omission rather than extrapolating an unreliable total\.

##### Efficiency measurement\.

The reported chunk\-512 headline uses a 96 GB H20\-class platform with PyTorch 2\.10, Transformers 5\.5\.4, bf16, and SDPA; the archived log records the device index but not the GPU model, so the hardware label follows the experiment registry\. It performs one unmeasured warmup and reports the median of three repetitions\. Full\-context prefill times the complete forward; CoMem times document Write, CPU retrieval, query Write, and Read through logits\. At 128k, full\-context prefill peaks at 89\.36 GB while CoMem peaks at 18\.26 GB\. Peak memory ismax\_memory\_allocated\. The separate clean decode control uses 20 greedy steps and verifies token\-level correctness\.

The depth\-Pareto cohort \(Table[9](https://arxiv.org/html/2607.28263#A1.T9)\) runs on one H20 with the same bf16/SDPA harness and reports medians of three measurements\. It fixes the retrieved chunk IDs, top\-12 pack \(about 6,657 tokens\), chunk size 512, and no LoRA acrossj∈\{0,6,9,12\}j\\in\\\{0,6,9,12\\\}\. Its online Read and uncached packed\-step columns isolate the cost of replaying 36/30/27/24 upper layers\. The latter intentionally reruns the pack per step and must not be compared to the deployed resumed\-band KV\-cache decode below\. Offline Write is measured separately because cached states are reused across future queries\.

### A\.4Efficiency details

Table[3](https://arxiv.org/html/2607.28263#S5.T3)in the main paper reports full\-write\-inclusive prefill, resumed\-band decode, peak accelerator memory, and correctness; Table[4](https://arxiv.org/html/2607.28263#S5.T4)separately reports same\-platform LoRA overhead\. Cross\-platform timing ratios are not mixed\.

### A\.5Cross\-scale and MoE generality

The Qwen3 depth sweep in Table[7](https://arxiv.org/html/2607.28263#A1.T7)spans 0\.6B, 1\.7B, 4B, 8B, 14B, and 32B: content depth remains near0\.45​L0\.45L, while the zero\-shot readable boundary moves deeper with scale\. Table[22](https://arxiv.org/html/2607.28263#A1.T22)then evaluates a RULER\-derived conservative integer split for each frozen backbone on all 1,986 LoCoMo questions\. The no\-chat score improves strongly overall \(11\.48→\\to51\.81\) but is not strictly monotone\. In particular, inspection of the 14B generations shows unusually long chain\-of\-thought trajectories without the chat template: reasoning consumes much of the generation budget before the final answer, plausibly explaining its dip below 8B\. The chat template mitigates this behavior \(28\.80→\\to39\.58\), while it slightly reduces 8B and 32B; we therefore treat the chat column as a generation\-protocol sensitivity analysis rather than a second scaling curve\. Table[23](https://arxiv.org/html/2607.28263#A1.T23)adds Qwen3\-30B\-A3B, where Dense OOMs at 128k but CoMem runs at 63\.0 GB and scores 100 on RULER single needle\.

Table 22:Frozen CoMem transfers across the Qwen3 family on LoCoMo\.Each integer split is a conservative readout\-safe point selected from RULER, independently of LoCoMo; it is not the 50% readout\-crash boundary in Table[7](https://arxiv.org/html/2607.28263#A1.T7)\. Values are full\-set GPT\-4o Judge accuracy \(%\) over all 1,986 questions\. Both protocols use iterative BM25 \(top\-12, hop width 4\), 512\-token chunks, a BOS sink, greedy decoding, and a 512\-token generation cap; the chat column is a template\-sensitivity control\. The 14B no\-chat dip reflects unusually long chain\-of\-thought trajectories that delay the final answer; the chat template mitigates this behavior\. Scores improve strongly with scale but are not strictly monotone\.Table 23:CoMem onQwen3\-30B\-A3B\(48\-layer sparse MoE; prefill speedup==Dense/CoMem prefill time\)\. CoMem peak memory is context\-independent \(63\.063\.0GB across 8k/32k/128k\) while the full\-context Dense reference grows and*OOMs*at 128k on an NVIDIA H20\. At 128k CoMem still runs and scores RULERniah\_single100100\(Dense OOM→\\toCoMem100100\); its prefill stays<0\.7<0\.7s at every length\. The depth partition, retrieval, and constant read thus hold at sparse\-MoE scale\.#### A\.5\.1Hunyuan Hy3: an 80\-layer sparse MoE

We additionally port CoMem to Tencent Hunyuan Hy3, an 80\-layer sparse Mixture\-of\-Experts model \(192 experts, top\-8 routing plus one shared expert;≈597\\approx 597GB in bf16\) with a 262,144\-token native window\. The backbone is sharded across eight GPUs; Write and Read semantics are unchanged\.

##### Exact partition through MoE routing\.

Reconstructing a contiguous forward by writinglayers\[0:jj\]and resuminglayers\[jj:80\]givesmax⁡\|Δ​logit\|=0\.0\\max\|\\Delta\\mathrm\{logit\}\|=0\.0atj∈\{0,1,40,80\}j\\in\\\{0,1,40,80\\\}, including the Dense\-to\-MoE boundary\. The resumed half re\-executes every router and expert, so the partition preserves routing exactly on this self\-test\.

##### Split depth and self\-distillation\.

A PG19 sweep places the best readout region nearj≈32j\\approx 32–36\. Atj=32j=32, self\-distillation reduces the 16k multiplicative LM tax from 1\.496 to 1\.078 and raises top\-1 agreement from 0\.773 to 0\.871 \(Table[24](https://arxiv.org/html/2607.28263#A1.T24)\)\.

Table 24:CoMem self\-distillation on Hy3 at splitj=32j\{=\}32\(LoRAr=32,α=64r\{=\}32,\\alpha\{=\}64onlayers\[32:\], teacherj=0j\{=\}0RAG recompute, 4000 steps of plain PG19 KL, no synthetic data\)\.gapis the multiplicative LM tax \(CoMem\-readout perplexity / full\-context perplexity;1\.01\.0is exact\);top1is argmax agreement with the full forward, over matched 16\-doc PG19 windows\. The 16k LM tax falls from\+49\.6%\+49\.6\\%to\+7\.8%\+7\.8\\%and top1 rises0\.773→0\.8710\.773\\to 0\.871, reproducing the 8B\-backbone self\-distillation effect on the 80\-layer MoE\.
##### Bounded read through 256k\.

With the distilled adapter and BM25 top\-8, the read stays at≈4\.3\\approx 4\.3–4\.6k tokens from 16k through 256k\. CoMem reaches 100/98 single/multikey recall at the 256k native ceiling with no OOM \(Table[25](https://arxiv.org/html/2607.28263#A1.T25)\)\.

Table 25:CoMem on the Tencent HunyuanHy3backbone \(hy\_v3, 80\-layer sparse MoE\), RULER needle recall \(n=50n\{=\}50, officialstring\_match\_all\), splitj=32j\{=\}32, BM25 top\-88, self\-distilled LoRA\. The read length fed to the model stays≈\\approx4\.3–4\.6k tokens while the context grows16×16\\times\(16k→\\to256k\); CoMem retains 100/98 recall at the backbone’s 256k native ceiling, with0out\-of\-memory failures\. The context row reports the mean packed length of the RULER samples\.

## Appendix BChat\-Template\-Free Statistical Verification

All statistics in this appendix were recomputed from the saved prediction shards on the shared result filesystem, without GPU reruns\. The Qwen3\-8B flagship uses the self\-distilledj=12j\{=\}12adapter, chunk 512, top\-12 retrieval, a BOS sink, and no chat template\. The adapter artifact isoutputs/qcmem\_distill\_qwen\_j12\_r32\_4k/final; this internal path is reported only for reproducibility\. The main RULER directory uses theiter\_bm25implementation; the saved configuration recordsrounds=0and an observed read of approximately 6\.5k tokens\. Every reported RULER cell contains eight of eight shards, 100 examples, no empty prediction, and zero mismatch between the stored score and a fresh official\-kernel recomputation\. BABILong likewise contains 100 valid examples in every cell\.

### B\.1RULER headline definition

Table 26:The 15 RULER cells entering the 97\.05 headline\. The score is their unweighted arithmetic mean:1455\.8/15=97\.05331455\.8/15=97\.0533\. The separately evaluated 256k cells are excluded from this headline\.Table[27](https://arxiv.org/html/2607.28263#A2.T27)reports the separate 256k extension\. Those three cells are excluded from the 97\.05 headline to keep its predefined 8k–128k support\.

Table 27:CoMem RULER scaling without a chat template \(n=100n\{=\}100per cell\)\. The 97\.05 headline averages only the 15 cells from 8k through 128k; 256k is a separately reported extension and is excluded from that headline\.
### B\.2LoCoMo judge protocol and denominator

Prediction shards for all methods and the adapter\-free CoMem control deduplicate to all 1,986 LoCoMo items\. GPT\-4o judges the 1,540 answerable items in categories 1–4\. Category 5 contains 446 adversarial abstention items and is scored locally: an answer is correct iff it is empty or matches the refusal rule\. Thus the canonical full\-set judge is over all 1,986 items, not only the API\-judged subset\.

Table 28:LoCoMo GPT\-4o judge by category\. Categories 1–4 contain 282/321/96/841 judged items; Category 5 contains 446 locally scored abstention items\. Overall aggregates all 1,986 examples\.
The judge uses the OpenAI\-compatible endpoint[https://maas\-openapi\.wanjiedata\.com/api/v1/chat/completions](https://maas-openapi.wanjiedata.com/api/v1/chat/completions)with model namegpt\-4o,seed=1, and no client\-set temperature or top\-pp\. The endpoint does not expose a dated model snapshot\. Requests are retried four times with exponential backoff; unparsable responses and API failures are conservatively scored wrong\. The prompt presents the question, gold answer, and prediction; it accepts any semantic match \(including equivalent date phrasing\), rejects contradictions, omissions, empty answers, and refusals, and requires exactlyCORRECTorWRONG\. The verbatim template is released with the evaluation script\. Gold alternatives are joined with “OR\.” A reply must begin withCORRECTorWRONG; a fallback substring vote is used before defaulting to wrong\.

On the common GPT\-4o\-judged items \(n=1540n\{=\}1540\), CoMem scores 48\.64 and KV\-Direct 43\.83\. A paired per\-item bootstrap of their difference \(10,000 resamples, seed 1234\) gives\+4\.81\+4\.81points with 95% CI\[2\.34,7\.27\]\[2\.34,7\.27\]\. Because the questions are nested within only 10 conversations, this interval is not our sole inferential check\. A paired conversation\-cluster bootstrap, resampling the 10 conversations rather than individual questions, also yields a 95% interval entirely above zero; 8 of the 10 observed conversation\-level differences favor CoMem\. We report the latter as a dependence\-aware robustness check rather than claiming that 1,540 questions are independent\. The canonical full\-set point scores are 38\.27 and 34\.59, respectively\.

### B\.3Independent\-judge audit

We reran the binary prompt with DeepSeek\-V3 on a stratified 200\-item subset shared by CoMem and KV\-Direct\. Agreement with GPT\-4o is 0\.81 \(κ=0\.626\\kappa=0\.626\)\. Both judges preserve the ordering: GPT\-4o scores 36\.5/32\.5 \(\+4\.0\+4\.0\) and DeepSeek\-V3 56\.0/49\.0 \(\+7\.0\+7\.0\) for CoMem/KV\-Direct\. Absolute calibration differs, but the positive gap is judge\-robust\.

### B\.4Uncertainty and aggregation checks

Per\-method, 1,000\-resample item\-bootstrap intervals \(seed 1234\) are\[36\.20,40\.58\]\[36\.20,40\.58\]for CoMem’s full\-set LoCoMo judge,\[32\.48,36\.71\]\[32\.48,36\.71\]for KV\-Direct’s,\[46\.04,50\.98\]\[46\.04,50\.98\]and\[41\.23,46\.30\]\[41\.23,46\.30\]for their category\-1–4 judge scores, and\[8\.60,9\.72\]\[8\.60,9\.72\]and\[8\.51,9\.61\]\[8\.51,9\.61\]for their F1 scores\. These unpaired, item\-level intervals are descriptive; the paired cluster analysis above is the appropriate dependence\-aware comparison\.

Selected 100\-example cell intervals \(1,000 resamples, seed 2024\) are: variable tracking 64k, 99\.0\[98\.2,99\.8\]\[98\.2,99\.8\], and 128k, 95\.8\[94\.2,97\.4\]\[94\.2,97\.4\]; single needle 32k, 99\.0\[97,100\]\[97,100\]; and multikey 64k, 91\.0\[85,96\]\[85,96\]\. We omit legacy BABILong intervals computed from an earlier prediction cohort\. LongEval scores 72\.83 over 600 examples \(95% CI\[69\.33,76\.33\]\[69\.33,76\.33\]\)\.