A “knowledge hard drive” that can be plugged and played across models—the difficulty lies not in moving the memory table itself, but in properly configuring the read head for the target model. https://arxiv.org/abs/2608.17050 The paper first trains an Engram hashed memory using the source model, then freezes both the memory and the target backbone, training only the reader to perform addressing, hidden space alignment, and specified layer injection. A dual-layer, four-branch reader achieves 38.8 on controlled QA, and gains are also seen in cross-architecture transfer. This is highly valuable for Agent memory: knowledge can be updated, audited, and reused without the context overhead of RAG. The limitation is that it was only tested up to 9B; scaling laws and joint training effects remain unknown. —

Cross-Model Memory Transfer via Target-Side Reader Adaptation

Source: https://arxiv.org/html/2608.17050

Mingyuan Li† Affiliation: ELLIS Institute Finland, University of Turku Emails: [email protected], [email protected], [email protected], [email protected]

Guangsheng Yu Affiliation: University of Technology Sydney Emails: [email protected], [email protected], [email protected], [email protected]

Xu Wang Affiliation: University of Technology Sydney Emails: [email protected], [email protected], [email protected], [email protected]

Shaoxiong Ji††thanks: Corresponding author. Affiliation: University of Turku Emails: [email protected], [email protected], [email protected], [email protected]

Abstract

Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access to external knowledge, but adds retrieval latency, context overhead, and only shallow integration with the backbone. Parametric adaptation is efficient at inference time, but entangles knowledge with model weights and can be hard to update, audit, or transfer. Engram-style hashed memory occupies a middle regime: it stores learned information in an external, addressable table, yet consumes that table through a small learned reader. This raises a basic question: when such a memory is moved across backbones, what matters more, the frozen memory itself or the target-side reader? We study this question through cross-model frozen-memory extraction, in which a memory trained on a source model is frozen and attached to a different target model, with only a lightweight reader trained. Ablations show that learned memory content and correct addressing both matter, but the transferred table becomes useful only through a reader aligned to the target model. In downstream question answering tasks, a dual-layer, four-branch reader nearly closes the gap between same-model and cross-model reuse, achieving an average score of 38.8 under our controlled evaluation protocol. Moreover, when the provider reader is directly compatible with the target interface, the frozen artifact can provide substantial utility without target-side training, while optional reader adaptation yields further improvement. These results suggest that Engram can serve as a reusable external knowledge artifact, provided that the target has access to a compatible reader interface; target-side adaptation can further improve alignment when direct reader reuse is insufficient.

1 Introduction

Language models can use knowledge in two familiar ways. Non-parametric methods, such as retrieval-augmented generation (RAG)Lewis et al., 2020 and nearest-neighbor retrievalKhandelwal et al., 2020, keep knowledge outside the backbone and make updates relatively direct, but they pay for that flexibility with retrieval latency, longer contexts, and loose integration between retrieved evidence and generation. Parametric methods, such as continued pretraining, fine-tuning, or learned memory layers tied to a single modelWei et al., 2026, avoid retrieval at inference time, but they entangle knowledge with model weights. This makes updates, audits, and transfers across backbones difficult. Engram-style hashed memoryCheng et al., 2026 occupies an interesting middle ground. It stores learned information in an explicit, addressable external table, yet consumes that table through a small neural interface rather than by inserting raw documents into the context. This hybrid design is attractive because it combines the modularity of external memory with the efficiency of learned representations. But it also exposes a sharper question: is the memory itself reusable or transferable, or is it merely a co-adapted extension of the model that trained it? The distinction matters: a memory table that is not portable is just another form of model-specific parameterization, not a reusable knowledge substrate. Thus, portability requires an operational test: remove the source backbone and ask whether another model can still extract a useful signal from the frozen memory table.

We study this test through cross-model frozen-memory transfer. Given a memory table trained with a source model A, we freeze the table, attach it to a different target model B, and train only a lightweight target-side reader for memory extraction. If the frozen memory still improves the target after the original host is removed, then it contains usable information beyond source-specific co-adaptation. If it fails, then the apparent memory benefit was likely tied to the original backbone. Figure 1 illustrates the lifecycle of memory reuse and transfer: source-side memory training, memory export and freezing, and target-side extraction through a new reader. To make this test precise, we separate three roles: (1) addressing decides which entries are read, (2) memory denotes the stored vectors, and (3) reader maps them into the model. We standardize addressing, freeze the memory, and adapt only the reader. This turns portability into a measurable question: can a target model learn to read a fixed memory artifact? We use a tokenizer-agnostic canonicalization pipeline to keep memory addresses stable across tokenizers, freeze the exported source memory, and train only a small target-side reader that projects memory values into the target residual stream.

Our central claim is: for portable hashed external memory, memory presence alone is not enough; successful reuse depends on the interface through which a target backbone addresses and consumes the stored representations. Nevertheless, memory content still matters. Transferred memory clearly outperforms permuted-key controls and reaches substantially better downstream Question Answering (QA) performance than random memory under the final evaluation protocol. But the amount of usable signal depends on how well the target model can align and integrate what it retrieves. This view explains why imperfect memory can still help with a capable reader, and why reader design can matter as much as retraining or enlarging the stored table in this regime.

We support this claim with a concise chain of evidence, spanning diverse model families and settings. First, a source–target transfer study shows that frozen memory consistently improves target models across different model families and scales, including PythiaBiderman et al., 2023, QwenQwen Team [2025], TinyLlamaZhang et al., 2024, PhiAbouelenin et al. [2025], LLaMATouvron et al., 2023, and MistralJiang et al., 2023, with relative perplexity reductions of up to 15.7%. A supplementary peer-to-peer setting further confirms that transfer remains beneficial in both directions between models of similar scale. Second, ablation experiments show that transferred memory clearly outperforms the permuted-key and no-memory controls, while scratch-trained target memory remains competitive under the same target-side adaptation budget. This suggests that memory reuse is a joint property of the stored content and the target-side reader that makes it usable. Third, on downstream tasks, upgrading the target-side reader to a dual-layer, multi-branch architecture nearly closes the gap between same-model and cross-model reuse, with the best configuration reaching an average score of 38.8 across five QA tasks, achieving a new state-of-the-art result (SoTA) and outperforming MLP MemoryWei et al., 2026 and selective gains on other knowledge-intensive benchmarks.

In summary, our contributions are:

• • We formulate cross-model frozen-memory reuse as an evaluation problem for external memory, separating portable memory from backbone-specific co-adaptation and measuring whether the target model can extract usable signal from a frozen artifact.
• • We instantiate this problem with a transfer protocol for Engram-style memory that standardizes addressing across tokenizers, and we propose lightweight target-side readers for memory extraction, including stronger multi-branch, multi-layer variants.
• • We provide an evidence chain showing that reader design is first-order: consistent gains across diverse model families and settings, ablations that isolate memory content and address integrity, and QA results showing that stronger target-side readers, rather than source-backbone identity alone, drive the best performance and achieve a new SoTA.

This paper is organized as follows. Section 2 reviews Engram architecture, related memory mechanisms, and cross-model alignment. Section 3 describes the cross-model frozen-memory transfer protocol, including tokenizer-agnostic addressing, target-side reader design, and training regime. Section 4 presents the experimental setup, research questions, results, and takeaways. Supplementary material in Appendix A 1 provides additional details on canonicalization, memory export, and the reader implementation. Appendix B 2 gives training and evaluation details, including hyperparameters, datasets, and evaluation metrics. Appendix C 3 extends RQ1 in Section 4.1 1 with exact matrix values and controls for tokenizer mismatch, peer-scale transfer, target scaling, and representational similarity. Appendix D 4 extends RQ3 in Section 4.3 3 by diagnosing why downstream gains are task-dependent, separating source-corpus specialization, Phase-2 reader-fitting alignment, mixed-corpus behavior, and out-of-domain effects. The supplementary material concludes with a computational cost analysis (Appendix F 6), broader ethics and impact considerations (Appendix G 7), and limitations (Appendix H 8).

2 Background and Related Work

2.1 Engram Architecture

EngramCheng et al., 2026 augments a Transformer-MoE backbone with an external conditional memory. It retrieves static n-gram embeddings through deterministic hashing and injects the retrieved vector into the backbone through a learned gate. The key property for our setting is that Engram separates addressing, storage, and read: the input text determines which memory entry is read, the memory table stores the external vectors, and a small learned reader maps the retrieved vector into the backbone hidden space. This separation makes transfer possible: as long as another model can reproduce the same memory addresses through a shared addressing scheme, the frozen Engram table can be reused through a target-side reader.

For each n-gram order ( n \in [2, N] ) and hash head ( k \in [1, K] ), a deterministic hash ( \varphi_{n,k} ) maps the canonicalized n-gram ( g_{t,n} ) to an entry in memory table ( \mathbf{E}{n,k} ). The retrieved embeddings are concatenated as ( \mathbf{e}{t} \triangleq \bigoplus_{n=2}^{N} \bigoplus_{k=1}^{K} \mathbf{E}{n,k}[\varphi{n,k}(g_{t,n})] ). Crucially, this lookup depends on the canonicalized input sequence rather than the model hidden state, so the memory table can be treated as an external artifact once the address space is standardized. The retrieved vector is consumed through learned projections and a gate: [ s_t = \frac{\text{RMSNorm}(\mathbf{h}{t})^\top \text{RMSNorm}(\mathbf{k}{t})}{\sqrt{d}}, \quad \alpha_t = \sigma\left(\text{sign}(s_t)\sqrt{|s_t|}\right), ] where ( \mathbf{k}{t} = \mathbf{W}K \mathbf{e}{t} ) and ( \mathbf{v}{t} = \mathbf{W}V \mathbf{e}{t} ). Here, ( \mathbf{W}_K ), ( \mathbf{W}_V ), and the gate define the backbone-specific reader. This structural separation makes frozen-memory transfer possible: the source memory table can be frozen and moved to a target model, while only a lightweight target-side reader is trained to extract and align it.

2.2 Memory Mechanisms

MoEShazeer et al., 2017, Fedus et al., 2022, Dai et al., 2024 expands capacity through conditional computation, but its useful signal remains embedded in the host backbone rather than exposed as a reusable memory artifact. Other memory-augmented models also miss the extraction question for one of two reasons: they either retrieve from large dynamic memory banks at inference timeGraves et al., 2014, Weston et al., 2015, or they store information in model-specific hidden spaces, as in KNN-LMKhandelwal et al., 2020. Retrieval-augmented generation and RETRO-style systemsLewis et al., 2020, Borgeaud et al., 2022, Izacard et al., 2023 are portable at the document level, but the reusable object is a text datastore rather than a trained memory representation. Larger learned memory layersLample et al., 2019, Berges et al., 2025, Behrouz and Zhong, 2025, Wei et al., 2026 improve capacity or efficient knowledge access, but their stored representations are usually trained and consumed within the same model. Engram differs in the regime relevant here: deterministic hashing provides an explicit address space, the memory table is external and addressable, and a small learned reader mediates read by the backbone. This separation makes a frozen memory artifact well defined and allows portability to be tested as extraction rather than as additional model capacity. A broader qualitative comparison appears in Table 7.

2.3 Cross-Model Alignment

Cross-model transfer is often studied as representation or adapter alignment. Embedding-space work suggests that independently trained models can sometimes be connected by simple linear maps between hidden representationsMikolov et al., 2013, Kornblith et al., 2019, Huh et al., 2024. Our setting adds a twist: rather than aligning activations, we freeze the memory table itself and let the target model learn a small projection to consume it. This turns alignment into a concrete reader-design problem.