Metis: Memory Foundation Model

# \titlefont Metis: Memory Foundation Model
Source: [https://arxiv.org/html/2607.26760](https://arxiv.org/html/2607.26760)
\\useunder

\\ul1\]MemTensor \(Shanghai\) Technology Co\., Ltd\. 2\]Renmin University of China 3\]National University of Singapore 4\]Shanghai Jiao Tong University 5\]Tongji University

Ziliang Guo1∗Yihang Sun1,4∗Xichong Zhang1∗Xixuan Hao1Zehao Lin1Yang Zhang3Xiaoyan Zhao3Tong Shen1,4Bo Tang1Zhi\-Qin John Xu4Junchi Yan4Haofen Wang5Xu Chen2†Feiyu Xiong1Zhiyu Li1†Tat\-Seng Chua3\[\[\[\[\[[lizy@memtensor\.cn](https://arxiv.org/html/2607.26760v1/mailto:[email protected])[xu\.chen@ruc\.edu\.cn](https://arxiv.org/html/2607.26760v1/mailto:[email protected])

\(July 29, 2026\)

###### Abstract

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models\. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored\. In this paper, we take a first step toward this direction by introducingmemory foundation models, which empower foundation models with native memory capabilities\. We formalizenative memoryfrom two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation\. We show that native memory offers advantages in architecture, end\-to\-end optimization, and efficiency\. Based on this formulation, we proposeMetis, the first prototype of memory foundation models\. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention\. We construct large\-scale memory\-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid\-training\. The online memory maintenance of Metis is gradient\-free, and the memory update requires only a forward pass\. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation\. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors\. To facilitate future research on memory foundation models, we release our project and model checkpoints\.

## 1Introduction

In recent years, large foundation models have achieved rapid development, demonstrating significant performance across many aspects, such as language modeling\[zhao2023survey,minaee2024large\], code generation\[jiang2026survey,chen2021evaluating,roziere2023code\], and complex reasoning\[xu2025toward,wei2022chain,guo2025deepseek\]\. This provides a solid foundation for constructing AI agents, which enables them to handle more complex tasks\. Beyond the reasoning capabilities of foundation models, memory is another critical capability of AI agents, responsible for retaining past information and leveraging it to support future inference\[zhang2025survey,hu2025memory\]\. In most previous works, memory is implemented by a module external to foundation models, rather than being natively integrated into their architectures\[zhong2024memorybank,xu2026mem,packer2024memgptllmsoperatingsystems,li2025memos\]\. Representative approaches use Retrieval\-Augmented Generation \(RAG\) to retrieve relevant textual information and incorporate it into the prompt to facilitate inference\[zhong2024memorybank\]\.

However, external memory suffers from several limitations presented inFigure[1](https://arxiv.org/html/2607.26760#S1.F1)\. First, external memory is decoupled from backbones with separated targets and processing stages\[zhong2024memorybank,packer2024memgptllmsoperatingsystems\]\. External memory typically aims to construct an informative context as input, and backbones only perform conditional language modeling over the constructed context\. Therefore, external memory may not provide the most useful information to support the backbone inference, and the backbone may not utilize the memory optimally\. Second, end\-to\-end optimization is difficult for external memory because gradients cannot be effectively propagated through discrete memory operations\[zhang2025learn\]\. As a result, performing domain\-specific post\-training becomes highly challenging\. Although some RL\-based strategies\[zhang2026memrl,guo2026memfactory\]can partially alleviate this issue by optimizing memory operations with reward signals, they suffer from efficiency issues\. Finally, external memory requires additional explicit operations over the storage outside backbones, which inevitably increases the online inference latency\[zhang2025survey\]\.

To address the limitations of external memory, we introducememory foundation modelsthat empower large foundation models withnative memory\. It converts memory from an external module into an internal mechanism of backbones, directly involved in forward computation\. Specifically, memory foundation models can generate responses based on the input instructions and their native memory, with autonomous memory transformation\. We define the native memory from two critical aspects: ∙\\bulletNative Memory State\.Unlike traditional large foundation models, memory foundation models are natively stateful across multiple inferences, which can formulate, maintain, and utilize memory states inside backbones from prior inferences\. Their memory states are dynamically represented as part of the parameters of backbones, whose semantic spaces are aligned at the pre\-training or mid\-training stage\. ∙\\bulletNative Memory Procedure\.Unlike memory engineering, memory foundation models natively integrate memory procedures within their inferences\. Specific memory operations, such as remembering, forgetting, and updating, are accomplished autonomously alongside the backbone’s forward computation, which impacts the native memory state based on input instructions\.

Memory foundation models aim to internalize memory capability into the model’s forward computation\. The memory state can be represented as dynamic parameters of backbones, and memory procedures are executed through computation\. Therefore, like general foundation models, memory foundation models can be optimized and adapted to specific domains in a data\-driven manner through post\-training\. This transformation is similar to the evolution from large foundation models to large reasoning models\[guo2025deepseek\], where Chain\-of\-Thought \(CoT\)\[wei2022chain\]is natively integrated into inference to improve performance and efficiency\. In addition, because native memory procedures can be integrated into the model’s computation, they provide a foundation for improving the parallel efficiency of memory processing\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x1.png)Figure 1:From external memory to native memory\.In this paper, we implement the first prototype of memory foundation models, namedMetis\. We design a new model architecture that has a native memory state inspired by Fast Weight Programming \(FWP\)\[ba2016using\], which can be integrated into the backbone computation through memory attention\. Specifically, we propose the Metis blocks as the basic units for native memory\. Each of them primarily consists of a hyper memory block and a local memory block\. In addition, we empower Metis with native memory procedures by designing specific optimization objectives, including memory reconstruction and memory operation objectives\. These two objectives correspond to the compression upper limit of memory states and operation targets\. We also design a regularization objective to improve robustness in complex scenarios\. To support this training, we synthesize large\-scale memory\-specific data from publicly available datasets, enabling Metis to acquire native memory capabilities through mid\-training\. Finally, we conduct extensive experiments to demonstrate the effectiveness of our proposed framework, and explore more aspects for analysis\.

From a general perspective, a fundamental problem of memory results from the time\-streaming property of online information\. At the storage stage, memory systems cannot determine how the received information will be used in the future\. At the inference stage, the original information is no longer accessible, and only the stored information can be utilized\. Therefore, memory can be considered as a prediction problem, where the model predicts how received information will be utilized in the future\. Like other prediction tasks in machine learning, memory capability can also be acquired at the pre\-training stage and generalized to other domains, and memory foundation models can provide the architectural and optimization foundation\.

Despite their promising properties, implementing memory foundation models remains highly challenging because their final goal is to completely eliminate the reliance on external memory in contexts\. While Metis achieves great performance in memory\-related tasks, it still faces several limitations\. First, its performance degrades on long\-term tasks, due to the information loss when compressed into fixed\-size parameters\. Second, it exhibits information confusion in some cases, possibly caused by the blending of semantics within the latent space\. Despite these limitations, Metis provides a potential pathway to achieve memory foundation models\. To benefit both the research community and industry, we release our project at[https://github\.com/MemTensor/Metis](https://github.com/MemTensor/Metis)\.

Our contributions are summarized as follows: ∙\\bulletWe introduce memory foundation models and native memory with formal definitions, providing further analysis from the perspective of native memory state and native memory procedure\. ∙\\bulletWe propose the first prototype of memory foundation models, named Metis, which is implemented with novel memory architectures and optimization tasks\. ∙\\bulletWe conduct extensive experiments to verify the effectiveness of our model, followed by detailed studies from multiple perspectives\. We also publicly release our project to benefit the research community and industry\.

The rest of our paper is organized as follows\.Section[2](https://arxiv.org/html/2607.26760#S2)provides the formal definition of memory foundation models\.Section[3](https://arxiv.org/html/2607.26760#S3)details the model architecture of Metis\. After that, we introduce our data construction pipeline inSection[4](https://arxiv.org/html/2607.26760#S4)and outline the optimization inSection[5](https://arxiv.org/html/2607.26760#S5)\. Extensive experimental results and analysis are presented inSection[6](https://arxiv.org/html/2607.26760#S6)\. Finally, we review related work inSection[7](https://arxiv.org/html/2607.26760#S7)and conclude inSection[8](https://arxiv.org/html/2607.26760#S8)\.

## 2Memory Foundation Model

In this section, we provide a formal definition of the memory foundation model\. Then, we introduce native memory from the perspectives of the native memory state and procedure\. After that, we compare memory foundation models with previous works\. Finally, we further discuss memory foundation models from the perspectives of lifelong learning and the evolving trends of AI agents\.

### 2\.1Definition

We define the memory foundation model under the multi\-step scenario\. Let a continuous interaction process be formulated as a sequence of discrete time stepst∈\{1,2,…,T\}t\\in\\\{1,2,\\dots,T\\\}\. At each steptt, the foundation model receives an input instruction sequence denoted asXtX\_\{t\}and generates a response sequenceYtY\_\{t\}\.

For traditional foundation models without memory, the generation relies entirely on the current input context\. The autoregressive decoding of thekk\-th token in the response is typically expressed asyt,k∼P​\(y∣Xt,Yt,<k;θ\)y\_\{t,k\}\\sim P\(y\\mid X\_\{t\},Y\_\{t,<k\};\\theta\), whereYt,<kY\_\{t,<k\}denotes the previously generated tokens at steptt, andθ\\thetarepresents the fixed parameters of the backbone\. For foundation models with external memory, the autoregressive decoding process is then conditioned on the contextCtC\_\{t\}alongside the input instruction\. It can be expressed byyt,k∼P​\(y∣Ct,Xt,Yt,<k;θ\)y\_\{t,k\}\\sim P\(y\\mid C\_\{t\},X\_\{t\},Y\_\{t,<k\};\\theta\), whereCtC\_\{t\}can be obtained from prior information\{\(Xi,Yi\)\}i=1t−1\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{t\-1\}\. The external memory framework commonly has two explicit procedures, including the storage procedure𝐂t=𝐂t−1⊕\{\(Xt,Yt\)\}\\mathbf\{C\}\_\{t\}=\\mathbf\{C\}\_\{t\-1\}\\oplus\\\{\\left\(X\_\{t\},Y\_\{t\}\\right\)\\\}, and the retrieval procedureCt=𝐂t−1⊗XtC\_\{t\}=\\mathbf\{C\}\_\{t\-1\}\\otimes X\_\{t\}\. Here,⊕\\oplusdenotes the general writing operation, and⊗\\otimesrepresents the general reading operation with the textual memory storage𝐂t−1\\mathbf\{C\}\_\{t\-1\}\. Both of them are executed outside the model inference process\.

###### Definition 1\(Memory Foundation Model\)\.

The memory foundation model is defined as an autoregressive foundation model empowered bynative memoryacross multi\-step interactions\. At each steptt, the generation of thekk\-th token is conditioned on the input instructionXtX\_\{t\}and the previously generated tokensYt,<kY\_\{t,<k\}by

yt,k∼P​\(y∣Xt,Yt,<k;θt\),y\_\{t,k\}\\sim P\(y\\mid X\_\{t\},Y\_\{t,<k\};\\theta\_\{t\}\),where the model parameterθt\\theta\_\{t\}integrates information from previous steps into its native parametric space \(i\.e\.,native memory state\)\. Concurrently,θt\+1\\theta\_\{t\+1\}is autonomously transformed during the forward computation inside the model based on the input instructionXtX\_\{t\}and outputYtY\_\{t\}\(i\.e\.,native memory procedure\)\.

In contrast, the memory foundation model internalizes memory into the backbone’s computation, which is empowered with native memory\. Instead of relying on an external explicit storage𝐂t\\mathbf\{C\}\_\{t\}and contextCtC\_\{t\}, it maintains a native memory state, which acts as dynamic parameters across multiple steps\. In addition, rather than explicitly executing memory procedures outside backbones, memory procedures in the memory foundation model occur autonomously alongside the model’s forward computation, such as operations like remembering, forgetting, and updating\.

### 2\.2Native Memory State

In this paper, we adopt a strict definition for the source of memory\. We only consider the information acquired during online interactions as memory, where information available before the interaction starts is excluded\. In fact, such offline information is better viewed as knowledge rather than memory, because it does not contain trajectory\-specific information for personalization and does not require real\-time adaptation\.

Since the stored information varies across different steps, the parametersθt\\theta\_\{t\}cannot remain completely static\. Consequently, at least a portion of the parameters must change dynamically according to the input, and we denote this dynamic part as the memory state𝐌t\\mathbf\{M\}\_\{t\}\. In the memory foundation model, the representation of stored information is supposed to be coupled with the backbone to participate in forward computation\. Therefore, the native memory state should be represented in parametric form\. Although textual memory offers advantages in interpretability and cross\-model compatibility, its discrete representation results in low information density and requires repetitive prefilling\. In contrast, parametric memory represents prior information in a dense form, which increases the efficiency of storage and utilization\.

In addition, the semantic spaces of both dynamic parameters𝐌t\\mathbf\{M\}\_\{t\}and static parametersΦ=θt∖𝐌t\\Phi=\\theta\_\{t\}\\setminus\\mathbf\{M\}\_\{t\}must be aligned during the pre\-training or mid\-training stage before conducting online inference\. This alignment enables the dynamic parameters at different steps to compute collaboratively with the static parameters\. During online interactions, the native memory state can be updated and utilized through the native memory procedure, which empowers the memory foundation model with statefulness across different steps\.

### 2\.3Native Memory Procedure

In terms of memory, storage and utilization are two core procedures to handle online information with the time\-streaming property\. Memory storage retains past information, while memory utilization leverages this stored information to support model inference\. They aim to address the temporal mismatch between information supply and usage\.

The memory storage procedure typically involves several specific operations, such as remembering, forgetting, and updating\. From the perspective of foundation models, an input instruction contains both the intent and the content of information processing\. For instance,“Alice is 24 years old”implies remembering her age, while“Bob moved from London to Boston”indicates updating his location\. A native storage procedure should directly map the input instruction to the update value of the memory state\. In contrast, external memory relies on rule\-based and predefined operations to handle its intent and content separately\. Although most operations can be categorized into insertion, deletion, and modification, the semantic intent and content cannot be easily decoupled into discrete rules\. In fact, the storage procedure essentially predicts how current information will be used in the future\. Because rule\-based procedures operate in a discrete function space, they struggle to achieve optimal prediction performance\.

The primary goal of the memory utilization procedure is to assist inference with the stored information\. From the foundation model perspective, it can be considered as letting the required information of the input instruction participate in the forward computation\. For example, answering“Where does Bob live now?”requires previously stored living information to facilitate inference\. Thus, a native memory utilization procedure should directly map the input instruction and memory state to the generated output\. External memory designs rules to trigger retrieval, reranking, and concatenation\. However, the information requirement cannot be defined and captured by discrete and finite rules\. For example, an instruction may require information based on semantic similarity, emotion, or even complex combinations of implicit metrics\. The memory utilization procedure predicts the information requirements, which is coupled with the inference process\. Therefore, it should not be divided into discrete stages limited by discrete function spaces\.

Consequently, the memory procedure should be modeled within a continuous function space and implemented via numerical computation, which is tightly coupled with the forward computation of the backbone\. In a memory foundation model, the native memory procedure autonomously executes both memory storage and utilization during the forward computation\. This native memory procedure should be established during the pre\-training or mid\-training stage\. In addition, this memory procedure paradigm has significant advantages in both efficiency and end\-to\-end optimization\.

### 2\.4Comparison with Previous Works

Test\-time Training\.Memory foundation models differ from test\-time training \(TTT\) in three key aspects\. First, TTT typically adapts the model within a single sequence, where the dynamic parameters are updated to better fit the current input\. In contrast, memory foundation models are defined under multi\-step interactions\. Their dynamic parameters serve as persistent native memory states that store information from previous steps and support future inference\.

Second, TTT does not explicitly provide native memory procedures\. Its update is usually driven by self\-supervised language modeling, which helps the model absorb prior information within the current sequence\. However, it does not specify how the model should semantically remember, forget, update, or reflect on information according to input instructions\. In contrast, memory foundation models are trained with memory reconstruction and operation objectives, enabling the model to autonomously execute semantic memory operations in the latent parametric space and transform the native memory state accordingly\.

Third, many TTT methods are motivated by efficient long\-context modeling, and they often introduce recurrent layers to replace full attention\. However, memory foundation models pursue a different goal\. They do not aim to replace the standard full\-attention computation within the current step\. Instead, they introduce information from previous interaction steps through native memory states as residuals\.

In summary, TTT is primarily a mechanism for inference\-time adaptation, while memory foundation models formulate memory as a persistent, instruction\-driven, and procedure\-aware capability of foundation models\.

Memory\-Augmented Neural Networks\.Memory\-augmented neural networks \(MANNs\) introduce additional memory modules to neural models, such as differentiable memory slots and learned read\-write operations\[graves2014neural\]\. These models show that neural networks can store external information and retrieve it for later computation\. Nevertheless, memory foundation models differ in how memory is integrated with the backbone\. In MANNs, the memory module is usually a separate storage component controlled by a neural controller\. Although the operations can be differentiable, the memory is still external to the main model parameters, which are often designed independently from the backbone\.

In contrast, memory foundation models internalize memory into the backbone computation\. The memory state is represented in a parametric form and participates directly in forward computation\. The memory procedure is also modeled by the same continuous function space as the backbone, rather than being implemented as a separate controller over explicit slots\. Therefore, memory foundation models can be regarded as a step from externally augmented memory toward native memory inside foundation models\.

Other Methods\.Compared with In\-place TTT\[feng2026place\], MemGen\[zhang2025memgen\]andδ\\delta\-Mem\[lei2026delta\], which still rely on textual memories in the context and use additional latent summaries to improve inference, memory foundation models remove textual memory from the context entirely\. Compared with MEMO\[quek2026memo\]and MemFT\[xu2026lora\], which primarily handle offline documents, memory foundation models focus on test\-time information\. Compared with Memory3\[yang2024memory3\], which takes an important step beyond textual RAG by encoding knowledge into retrievable explicit memories, memory foundation models further extend this direction toward native memory\. While Memory3primarily constructs explicit memories from offline corpora and retrieves them to augment attention computation, memory foundation models internalize information acquired from online interactions as persistent dynamic states within the backbone\. They further enable these states to be autonomously maintained and transformed through native memory procedures across multiple interaction steps\.

### 2\.5Discussion

For memory foundation models, the onset of interaction represents a key transition from static to dynamic knowledge acquisition\. Knowledge acquired before interaction originates from offline pre\-training and is retained in static parameters, while information received during interaction is acquired at test time and stored in dynamic memory states\. Therefore,θ1\\theta\_\{1\}can also serve as initial supplementary information outside of pre\-training\. Becauseθ1\\theta\_\{1\}captures transferable domain knowledge, models deployed in similar domains can be initialized with the sameθ1\\theta\_\{1\}to provide baseline information\.

Moreover, native memory aligns with the evolving trend of foundation models\. Inspired by large reasoning models\[guo2025deepseek\], we find that an agent’s external capabilities can be expressed natively by the foundation model through optimization\. In other words, supervised data of target behaviors can activate internal capabilities and generalize them to other tasks\. These native capabilities can provide advantages in generalization, efficiency, and optimization properties\. Memory is also a critical agent capability that traditionally relies on external modules\. Therefore, we argue that memory can also be natively triggered through memory\-specific tasks\. However, unlike reasoning, which is purely a process, memory also involves a storage entity\. It requires us to modify the model architecture to incorporate a storage entity as the memory state\. Then, both the memory state and procedures are supposed to be modeled under a collaborative function space\. This enables us to empower foundation models with memory capabilities via an optimization\-driven approach\.

## 3Metis Architecture

In this section, we first present some preliminaries\. Then, we introduce Metis as the prototype of memory foundation models with the Metis block\. After that, we demonstrate how this architecture supports native memory storage and utilization procedures through computation\. Finally, we provide theoretical insights, theoretical error analysis, and further discussions\. The overview of the Metis framework is presented inFigure[2](https://arxiv.org/html/2607.26760#S3.F2)\.

### 3\.1Preliminaries

We adopt causal language models as the primary implementation of memory foundation models, as they are dominantly used in modern foundation models\. We present the standard architecture of causal language models, which primarily consists ofNNstacked Transformer blocks followed by a language modeling head\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x2.png)Figure 2:Overview of the Metis architecture\.Transformer Block\.To highlight the core architecture, we focus on causal self\-attention and the feed\-forward network \(FFN\), which are major components of modern Transformers\. We omit other details, such as positional embeddings, multi\-head attention strategies, and hybrid attention mechanisms, as they can be directly incorporated into our framework\.

We denote the input of thell\-th Transformer block as𝐇\(l−1\)∈ℝL×d\\mathbf\{H\}^\{\(l\-1\)\}\\in\\mathbb\{R\}^\{L\\times d\}, whereLLis the sequence length andddis the dimension of hidden states\. After applying the pre\-normalization functionPreNorm, we obtain𝐇~\(l\)=PreNorm​\(𝐇\(l−1\)\),\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}=\\text\{PreNorm\}\(\\mathbf\{H\}^\{\(l\-1\)\}\),as the input of causal self\-attention\. We denote𝐖Q\(l\),𝐖K\(l\)∈ℝd×dk,𝐖V\(l\)∈ℝd×dv\\mathbf\{W\}\_\{Q\}^\{\(l\)\},\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\},\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{v\}\}as query, key, and value projection matrices of this layer\. Then, we obtain the query state, key state, and value state of thell\-th layer by

𝐐\(l\)=𝐇~\(l\)​𝐖Q\(l\),𝐊\(l\)=𝐇~\(l\)​𝐖K\(l\),𝐕\(l\)=𝐇~\(l\)​𝐖V\(l\)\.\\mathbf\{Q\}^\{\(l\)\}=\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\\mathbf\{W\}\_\{Q\}^\{\(l\)\},\\quad\\mathbf\{K\}^\{\(l\)\}=\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\\mathbf\{W\}\_\{K\}^\{\(l\)\},\\quad\\mathbf\{V\}^\{\(l\)\}=\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\.After that, the causal self\-attention can be calculated by

𝐀\(l\)=Softmax​\(𝐐\(l\)​\(𝐊\(l\)\)⊤dk\+Mask​\(L\)\)​𝐕\(l\),\\mathbf\{A\}^\{\(l\)\}=\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}^\{\(l\)\}\(\\mathbf\{K\}^\{\(l\)\}\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{V\}^\{\(l\)\},\(1\)wheredkd\_\{k\}is the attention head dimension, andMask​\(L\)\\text\{Mask\}\(L\)is the causal mask defined asMask​\(L\)i,j=−∞\\text\{Mask\}\(L\)\_\{i,j\}=\-\\inftyifj\>ij\>i, and0otherwise\. Then, it adds the projected output attention to the residual after projection by

𝐇′⁣\(l\)=𝐇\(l−1\)\+𝐀\(l\)​𝐖O\(l\),\\mathbf\{H\}^\{\\prime\(l\)\}=\\mathbf\{H\}^\{\(l\-1\)\}\+\\mathbf\{A\}^\{\(l\)\}\\mathbf\{W\}\_\{O\}^\{\(l\)\},where𝐖O\(l\)∈ℝdv×d\\mathbf\{W\}\_\{O\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\}\. Finally,𝐇′⁣\(l\)\\mathbf\{H\}^\{\\prime\(l\)\}is passed through FFN with a residual connection to get thell\-th layer output by

𝐇\(l\)=𝐇′⁣\(l\)\+FFN​\(Norm​\(𝐇′⁣\(l\)\)\),\\mathbf\{H\}^\{\(l\)\}=\\mathbf\{H\}^\{\\prime\(l\)\}\+\\text\{FFN\}\(\\text\{Norm\}\(\\mathbf\{H\}^\{\\prime\(l\)\}\)\),where the activation𝐇\(l\)\\mathbf\{H\}^\{\(l\)\}is also the input of\(l\+1\)\\left\(l\+1\\right\)\-th layer\.

Causal Language Model\.We denote the sequence of input tokens asX=\(x1,x2,…,xL\)X=\(x\_\{1\},x\_\{2\},\\dots,x\_\{L\}\)\. A causal language model first maps these discrete tokens into continuous vector representations\. Let𝐄∈ℝ\|𝒱\|×d\\mathbf\{E\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}denote the token embedding matrix, where\|𝒱\|\|\\mathcal\{V\}\|is the vocabulary size\. The initial hidden state𝐇\(0\)∈ℝL×d\\mathbf\{H\}^\{\(0\)\}\\in\\mathbb\{R\}^\{L\\times d\}is obtained by extracting the corresponding embeddings and combining them with positional embeddings\. After that, this initial representation is processed sequentially through the stack ofNNTransformer blocks by

𝐇\(l\)=TransformerBlock\(l\)​\(𝐇\(l−1\)\),for​l=1,2,…,N\.\\mathbf\{H\}^\{\(l\)\}=\\text\{TransformerBlock\}^\{\(l\)\}\(\\mathbf\{H\}^\{\(l\-1\)\}\),\\quad\\text\{for \}l=1,2,\\dots,N\.Then, the final hidden state𝐇\(N\)\\mathbf\{H\}^\{\(N\)\}represents the contextualized input, and the language modeling head maps this final state back to the vocabulary space to predict the probability distribution for the next token\. This process is commonly modeled by a linear projection after normalization, followed by a softmax function

P\(⋅∣x≤i\)=Softmax\(Norm\(𝐇\(i\)\(N\)\)𝐖LM\),P\(\\cdot\\mid x\_\{\\leq i\}\)=\\text\{Softmax\}\(\\text\{Norm\}\(\\mathbf\{H\}\_\{\(i\)\}^\{\(N\)\}\)\\mathbf\{W\}\_\{\\text\{LM\}\}\),where𝐇\(i\)\(N\)∈ℝd\\mathbf\{H\}\_\{\(i\)\}^\{\(N\)\}\\in\\mathbb\{R\}^\{d\}is the final hidden vector at positionii, and𝐖LM∈ℝd×\|𝒱\|\\mathbf\{W\}\_\{\\text\{LM\}\}\\in\\mathbb\{R\}^\{d\\times\|\\mathcal\{V\}\|\}represents the projection matrix\. After samplingxi\+1∼P​\(xi\+1∣x≤i\)x\_\{i\+1\}\\sim P\(x\_\{i\+1\}\\mid x\_\{\\leq i\}\), this new token is appended to the sequence, and the model repeats the process until it decodes an end\-of\-sequence token or reaches the maximum length\.

During the pre\-training phase, the causal language model is optimized using the standard autoregressive next\-token prediction objective\. It minimizes the negative log\-likelihood of the training sequences by

θ=arg⁡minθ​∑X∈𝒟∑i=1\|X\|−1−log⁡P​\(xi\+1∣x≤i;θ\),\\theta=\\arg\\min\_\{\\theta\}\\sum\_\{X\\in\\mathcal\{D\}\}\\sum\_\{i=1\}^\{\|X\|\-1\}\-\\log P\(x\_\{i\+1\}\\mid x\_\{\\leq i\};\\theta\),where𝒟\\mathcal\{D\}represents the pre\-training corpus andθ\\thetaencompasses all the trainable parameters of the model\.

### 3\.2Native Memory State

To implement the native memory state, we propose the Metis blocks inside Transformer blocks inFigure[2](https://arxiv.org/html/2607.26760#S3.F2)\(b\), where each Metis block consists of a local memory block and a hyper memory block inFigure[2](https://arxiv.org/html/2607.26760#S3.F2)\(c\)\. The local memory blocks are responsible for maintaining the dense representation of prior information, while the hyper memory blocks construct parametric function spaces for native memory procedures to transform memory states\.

Local Memory Block\.Local memory blocks maintain the memory state at the current step, so we define a dense memory network𝐌\(l\)∈ℝdk×dv\\mathbf\{M\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\_\{v\}\}inside thell\-th local memory block\. At the steptt, we denote it as𝐌t\(l\)\\mathbf\{M\}^\{\(l\)\}\_\{t\}\. The model also maintains a query\-key normalization vector as𝐒t\(l\)∈ℝdk\\mathbf\{S\}^\{\(l\)\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{k\}\}\. Both𝐌\(l\)\\mathbf\{M\}^\{\(l\)\}and𝐒t\(l\)\\mathbf\{S\}^\{\(l\)\}\_\{t\}are dynamic parameters updated across different steps\. Specifically, we set𝐌1\(l\)=𝟎\\mathbf\{M\}^\{\(l\)\}\_\{1\}=\\mathbf\{0\}and𝐒1\(l\)=𝟎\\mathbf\{S\}^\{\(l\)\}\_\{1\}=\\mathbf\{0\}by default\.

Hyper Memory Block\.Hyper memory blocks are responsible for updating the dynamic parameters in local memory blocks based on the intermediate activations of the current inputXtX\_\{t\}and outputYtY\_\{t\}\. Each of them consists of static parameters obtained through mid\-training, which remain unchanged during interactions\. It serves as the parametric foundation of the native memory storage procedure\. Specifically, each hyper memory block is parameterized by several optimizable parameters\. First, it has a learnable importance vector𝐰~agg\(l\)∈ℝd\\tilde\{\\mathbf\{w\}\}\_\{\\text\{agg\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\}, which scores the intermediate activations for adaptive aggregation\. In addition, we set the memory key and value projection matrices𝐖~K\(l\)∈ℝd×dk\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}and𝐖~V\(l\)∈ℝd×dv\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{v\}\}, which map the selected hidden states into the memory keys and values for the local memory\. We also set the memory query projection matrix𝐖~Q\(l\)∈ℝd×dk\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}to reduce the error in the native memory utilization procedure\.

### 3\.3Native Memory Procedure

The native memory procedure consists of memory storage and utilization procedures, as we discuss inSection[2\.3](https://arxiv.org/html/2607.26760#S2.SS3)\. In the native memory storage procedure of our framework, hyper memory blocks update local memory blocks as part of the model computation, based on intermediate activations\. In the native memory utilization procedure, local memory blocks incorporate the current memory states into the forward computation\.

Native Memory Storage Procedure\.After completing steptt, we denote the input hidden states at thell\-th layer as𝐇t\(l−1\)\\mathbf\{H\}^\{\(l\-1\)\}\_\{t\}\. Then, the hyper memory block aggregates it into a compact representation through a learned adaptive aggregation\. Specifically, we first pre\-normalize the hidden states as𝐇~t\(l\)=PreNorm​\(𝐇t\(l−1\)\)\\tilde\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}=\\text\{PreNorm\}\(\\mathbf\{H\}^\{\(l\-1\)\}\_\{t\}\)and score each of theLLtokens with a learnable importance vector𝐰~agg\(l\)∈ℝd\\tilde\{\\mathbf\{w\}\}\_\{\\text\{agg\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\}, obtaining an importance distribution

𝐩t\(l\)=Softmax​\(𝐇~t\(l\)​𝐰~agg\(l\)τ\)∈ℝL,\\mathbf\{p\}\_\{t\}^\{\(l\)\}=\\text\{Softmax\}\\left\(\\frac\{\\tilde\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}\\tilde\{\\mathbf\{w\}\}\_\{\\text\{agg\}\}^\{\(l\)\}\}\{\\tau\}\\right\)\\in\\mathbb\{R\}^\{L\},whereτ\\tauis a temperature coefficient\. Then, we obtain a subset of positions based on the top\-ρ\\rho\. We sort these probabilities in descending order asp\(1\)≥p\(2\)≥⋯≥p\(L\)p\_\{\(1\)\}\\geq p\_\{\(2\)\}\\geq\\cdots\\geq p\_\{\(L\)\}, and keep the smallest prefix whose cumulative value reaches the thresholdρ\\rho\. The number of selected positions can be expressed by

Lt′=clip​\(min⁡\{k:∑r=1kp\(r\)≥ρ\},Kmin,L\),L\_\{t\}^\{\\prime\}=\\text\{clip\}\\left\(\\min\\Big\\\{k:\\sum\\nolimits\_\{r=1\}^\{k\}p\_\{\(r\)\}\\geq\\rho\\Big\\\},\\ K\_\{\\min\},\\ L\\right\),whereKminK\_\{\\min\}denotes the minimum number of selected positions\. TheseLt′L\_\{t\}^\{\\prime\}positions with the highest scores form the selected set𝒮t\(l\)\\mathcal\{S\}\_\{t\}^\{\(l\)\}, and we gather corresponding hidden states by

𝐇¯t\(l\)=𝚷t\(l\)​𝐇~t\(l\)∈ℝLt′×d,with​Lt′≪L,\\bar\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}=\\mathbf\{\\Pi\}\_\{t\}^\{\(l\)\}\\tilde\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}\\in\\mathbb\{R\}^\{L\_\{t\}^\{\\prime\}\\times d\},\\quad\\text\{with \}L\_\{t\}^\{\\prime\}\\ll L,where𝚷t\(l\)∈\{0,1\}Lt′×L\\mathbf\{\\Pi\}\_\{t\}^\{\(l\)\}\\in\\\{0,1\\\}^\{L\_\{t\}^\{\\prime\}\\times L\}, whose rows are the one\-hot indicators of𝒮t\(l\)\\mathcal\{S\}\_\{t\}^\{\(l\)\}\. Since the top\-ρ\\rhoselection is non\-differentiable, we adopt a straight\-through estimator that routes the gradients through the dense distribution𝐩t\(l\)\\mathbf\{p\}\_\{t\}^\{\(l\)\}, making the scorer𝐰~agg\(l\)\\tilde\{\\mathbf\{w\}\}\_\{\\text\{agg\}\}^\{\(l\)\}end\-to\-end trainable\. After that, we compute the projected memory key states and memory value states as

𝐊~t\(l\)=𝐇¯t\(l\)​𝐖~K\(l\),𝐕~t\(l\)=𝐇¯t\(l\)​𝐖~V\(l\)\.\\tilde\{\\mathbf\{K\}\}\_\{t\}^\{\(l\)\}=\\bar\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\},\\quad\\tilde\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\}=\\bar\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}\.Finally, the dense memory network is updated based on𝐊~t\(l\)\\tilde\{\\mathbf\{K\}\}\_\{t\}^\{\(l\)\}and𝐕~t\(l\)\\tilde\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\}by

𝐌t\+1\(l\)=λ​𝐌t\(l\)\+\(1−λ\)Lt′⋅𝐊~t\(l\)⊤dk​𝐕~t\(l\),\\mathbf\{M\}^\{\(l\)\}\_\{t\+1\}=\\lambda\\mathbf\{M\}^\{\(l\)\}\_\{t\}\+\\frac\{\(1\-\\lambda\)\}\{L\_\{t\}^\{\\prime\}\}\\cdot\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\tilde\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\},\(2\)whereλ\\lambdarepresents the discount factor\. In addition, the query\-key normalization vector can be updated by

𝐒t\+1\(l\)=λ​𝐒t\(l\)\+\(1−λ\)Lt′⋅𝐊~t\(l\)⊤​𝟏dk\.\\mathbf\{S\}^\{\(l\)\}\_\{t\+1\}=\\lambda\\mathbf\{S\}^\{\(l\)\}\_\{t\}\+\\frac\{\(1\-\\lambda\)\}\{L\_\{t\}^\{\\prime\}\}\\cdot\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{t\}^\{\(l\)\\top\}\\mathbf\{1\}\}\{\\sqrt\{d\_\{k\}\}\}\.\(3\)This native storage procedure is presented inFigure[2](https://arxiv.org/html/2607.26760#S3.F2)\(c\)\. Based on the constructed function space, we aim to internalize various memory operations into the model’s computation through optimization\. Specifically, the selection and projection provide the model with compression capabilities\. Meanwhile, semantic\-based computation enables memory instructions to be understood and applied within the latent space\. In practice, we find that replacing the linear update with a Gated Delta Network \(GDN\)\-based\[yang2025gated\]update obtains better performance, so Metis finally adopts the GDN\-based update \(GDU\) strategy\.Section[6\.3](https://arxiv.org/html/2607.26760#S6.SS3)andAppendix[C](https://arxiv.org/html/2607.26760#A3)compares the two implementations through ablation studies\.

Native Memory Utilization Procedure\.We define the memory attention as

𝐀~t\(l\)=diag​\(𝐐~t\(l\)​𝐒t\(l\)\)−1​𝐐~t\(l\)​𝐌t\(l\),\\tilde\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{diag\}\\left\(\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\mathbf\{S\}^\{\(l\)\}\_\{t\}\\right\)^\{\-1\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\mathbf\{M\}^\{\(l\)\}\_\{t\},\(4\)where𝐐~t\(l\)=𝐇~t\(l\)​𝐖~Q\(l\)\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}=\\tilde\{\\mathbf\{H\}\}\_\{t\}^\{\(l\)\}\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}denotes the memory query states with optimizable parameter𝐖~Q\(l\)∈ℝd×dk\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}\. In practice, we add an identity vector to the normalization denominator to prevent numerical overflow and improve numerical stability\. Then, the memory attention is integrated into the main branch of attention, and replacesEquation \([1](https://arxiv.org/html/2607.26760#S3.E1)\)with

𝐀t\(l\)=γ⋅Softmax​\(𝐐t\(l\)​\(𝐊t\(l\)\)⊤dk\+Mask​\(L\)\)​𝐕t\(l\)\+\(1−γ\)⋅Norm​\(𝐀~t\(l\)\),\\mathbf\{A\}^\{\(l\)\}\_\{t\}=\\gamma\\cdot\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}^\{\(l\)\}\_\{t\}\(\\mathbf\{K\}^\{\(l\)\}\_\{t\}\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{V\}\_\{t\}^\{\(l\)\}\+\(1\-\\gamma\)\\cdot\\text\{Norm\}\\left\(\\tilde\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}\\right\),\(5\)whereNorm​\(⋅\)\\text\{Norm\}\(\\cdot\)is applied to the memory readout to align its scale with the original attention branch,𝐊t\(l\),𝐕t\(l\)\\mathbf\{K\}^\{\(l\)\}\_\{t\},\\mathbf\{V\}^\{\(l\)\}\_\{t\}are input key states and value states at the current step, andγ∈\[0,1\]\\gamma\\in\[0,1\]balances the two branches\.

### 3\.4Theoretical Insight of Native Memory Procedures

We provide theoretical insights on how information from previous steps influences subsequent inference through Metis blocks\. At steptt, we prepend an additional virtual memory prefix𝐏t\(l\)∈ℝLp×d\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\in\\mathbb\{R\}^\{L\_\{p\}\\times d\}to the input𝐇~t\(l\)\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}of thell\-th attention layer, resulting in the augmented input

𝐇^t\(l\)=\[𝐏t\(l\)𝐇~t\(l\)\]\.\\hat\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}=\\begin\{bmatrix\}\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\\\ \\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}\\end\{bmatrix\}\.Then, we compute the corresponding query state𝐐^t\(l\)\\hat\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}as follows

𝐐^t\(l\)=𝐇^t\(l\)​𝐖Q\(l\)=\[𝐏t\(l\)​𝐖Q\(l\)𝐐t\(l\)\]\.\\hat\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}=\\hat\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{Q\}^\{\(l\)\}=\\begin\{bmatrix\}\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{Q\}^\{\(l\)\}\\\\ \\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\end\{bmatrix\}\.Similarly, we have the key states and value states

𝐊^t\(l\)=\[𝐏t\(l\)​𝐖K\(l\)𝐊t\(l\)\],𝐕^t\(l\)=\[𝐏t\(l\)​𝐖V\(l\)𝐕t\(l\)\]\.\\hat\{\\mathbf\{K\}\}^\{\(l\)\}\_\{t\}=\\begin\{bmatrix\}\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\\\ \\mathbf\{K\}\_\{t\}^\{\(l\)\}\\end\{bmatrix\},\\quad\\hat\{\\mathbf\{V\}\}^\{\(l\)\}\_\{t\}=\\begin\{bmatrix\}\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\\\ \\mathbf\{V\}\_\{t\}^\{\(l\)\}\\end\{bmatrix\}\.The attention output𝐀^t\\hat\{\\mathbf\{A\}\}\_\{t\}is then computed using a modified causal maskMask​\(Lp\+L\)∈ℝ\(Lp\+L\)×\(Lp\+L\)\\text\{Mask\}\(L\_\{p\}\+L\)\\in\\mathbb\{R\}^\{\(L\_\{p\}\+L\)\\times\(L\_\{p\}\+L\)\}by

𝐀^t=Softmax​\(𝐐^t\(l\)​𝐊^t\(l\)⊤dk\+Mask​\(Lp\+L\)\)​𝐕^t\(l\)\.\\hat\{\\mathbf\{A\}\}\_\{t\}=\\text\{Softmax\}\\left\(\\frac\{\\hat\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\hat\{\\mathbf\{K\}\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\_\{p\}\+L\)\\right\)\\hat\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\}\.Specifically, we divide the causal mask into four parts as follows

Mask​\(Lp\+L\)=\[𝟎Lp×Lp−∞Lp×L𝟎L×LpMask​\(L\)\],\\text\{Mask\}\(L\_\{p\}\+L\)=\\left\[\\begin\{array\}\[\]\{c\|c\}\\mathbf\{0\}\_\{L\_\{p\}\\times L\_\{p\}\}&\-\\infty\_\{L\_\{p\}\\times L\}\\\\ \\hline\\cr\\mathbf\{0\}\_\{L\\times L\_\{p\}\}&\\text\{Mask\}\(L\)\\end\{array\}\\right\],where𝟎L×Lp\\mathbf\{0\}\_\{L\\times L\_\{p\}\}allows the virtual memory prefix tokens to be visible to input tokens\. Then, we decompose the calculation of attention as

𝐀^t=\[Softmax​\(\(𝐏t\(l\)​𝐖Q\(l\)\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)𝟎Lp×LSoftmax∗​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)Softmax∗​\(𝐐t\(l\)​𝐊t\(l\)⊤dk\+Mask​\(L\)\)\]​\[𝐏t\(l\)​𝐖V\(l\)𝐕t\(l\)\],\\hat\{\\mathbf\{A\}\}\_\{t\}=\\begin\{bmatrix\}\\text\{Softmax\}\\left\(\\frac\{\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{Q\}^\{\(l\)\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)&\\mathbf\{0\}\_\{L\_\{p\}\\times L\}\\\\ \\text\{Softmax\}^\{\*\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)&\\text\{Softmax\}^\{\*\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\mathbf\{K\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\end\{bmatrix\}\\begin\{bmatrix\}\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\\\ \\mathbf\{V\}\_\{t\}^\{\(l\)\}\\end\{bmatrix\},whereSoftmax∗​\(⋅\)\\text\{Softmax\}^\{\*\}\\left\(\\cdot\\right\)denotes the global softmax function applied to the entire row\. Then, we retain the attention outputs corresponding to the non\-virtual tokens by

𝐀t\(l\)=Softmax∗​\(𝐐t\(l\)​𝐊t\(l\)⊤dk\+Mask​\(L\)\)​𝐕t\(l\)⏟Original Attention\+Softmax∗​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)​\(𝐏t\(l\)​𝐖V\(l\)\)⏟Memory Attention\.\\mathbf\{A\}^\{\(l\)\}\_\{t\}=\\underbrace\{\\text\{Softmax\}^\{\*\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\mathbf\{K\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{V\}\_\{t\}^\{\(l\)\}\}\_\{\\text\{Original Attention\}\}\+\\underbrace\{\\text\{Softmax\}^\{\*\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\}\_\{\\text\{Memory Attention\}\}\.\(6\)
Let𝐳orig\\mathbf\{z\}\_\{\\text\{orig\}\}and𝐳mem\\mathbf\{z\}\_\{\\text\{mem\}\}denote the partition items of the original attention and memory attention

𝐳orig=exp⁡\(𝐐t\(l\)​𝐊t\(l\)⊤dk\+Mask​\(L\)\)​𝟏L,𝐳mem=exp⁡\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)​𝟏Lp\.\\mathbf\{z\}\_\{\\text\{orig\}\}=\\exp\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\mathbf\{K\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{1\}\_\{L\},\\quad\\mathbf\{z\}\_\{\\text\{mem\}\}=\\exp\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\mathbf\{1\}\_\{L\_\{p\}\}\.\(7\)Then, we perform element\-wise division by its element\-wise sum to get the weighting matrices

𝚲orig=diag​\(𝐳orig⊘\(𝐳orig\+𝐳mem\)\),𝚲mem=diag​\(𝐳mem⊘\(𝐳orig\+𝐳mem\)\)=𝐈−𝚲orig\.\\mathbf\{\\Lambda\}\_\{\\text\{orig\}\}=\\text\{diag\}\\left\(\\mathbf\{z\}\_\{\\text\{orig\}\}\\oslash\\left\(\\mathbf\{z\}\_\{\\text\{orig\}\}\+\\mathbf\{z\}\_\{\\text\{mem\}\}\\right\)\\right\),\\quad\\mathbf\{\\Lambda\}\_\{\\text\{mem\}\}=\\text\{diag\}\\left\(\\mathbf\{z\}\_\{\\text\{mem\}\}\\oslash\\left\(\\mathbf\{z\}\_\{\\text\{orig\}\}\+\\mathbf\{z\}\_\{\\text\{mem\}\}\\right\)\\right\)=\\mathbf\{I\}\-\\mathbf\{\\Lambda\}\_\{\\text\{orig\}\}\.
Then,Equation \([6](https://arxiv.org/html/2607.26760#S3.E6)\)is equivalent to the equation with the normal Softmax function for each part:

𝐀t\(l\)=𝚲orig⋅Softmax​\(𝐐t\(l\)​𝐊t\(l\)⊤dk\+Mask​\(L\)\)​𝐕t\(l\)\+\(𝐈−𝚲orig\)⋅Softmax​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)​\(𝐏t\(l\)​𝐖V\(l\)\)\.\\mathbf\{A\}^\{\(l\)\}\_\{t\}=\\mathbf\{\\Lambda\}\_\{\\text\{orig\}\}\\cdot\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\mathbf\{K\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{V\}\_\{t\}^\{\(l\)\}\+\\left\(\\mathbf\{I\}\-\\mathbf\{\\Lambda\}\_\{\\text\{orig\}\}\\right\)\\cdot\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\.To control the influence of the two attention components, we introduce a global weighting parameterγ∈\[0,1\]\\gamma\\in\[0,1\]to approximate the original weighting matrices by

𝐀t\(l\)=γ⋅Softmax​\(𝐐t\(l\)​𝐊t\(l\)⊤dk\+Mask​\(L\)\)​𝐕t\(l\)\+\(1−γ\)⋅Softmax​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)​\(𝐏t\(l\)​𝐖V\(l\)\)\.\\mathbf\{A\}^\{\(l\)\}\_\{t\}=\\gamma\\cdot\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\mathbf\{K\}\_\{t\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+\\text\{Mask\}\(L\)\\right\)\\mathbf\{V\}\_\{t\}^\{\(l\)\}\+\(1\-\\gamma\)\\cdot\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\.Then, we denote this specific memory attention part for𝐏t\(l\)\\mathbf\{P\}^\{\(l\)\}\_\{t\}as

𝐀ˇt\(l\)=Softmax​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)​\(𝐏t\(l\)​𝐖V\(l\)\)\.\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{Softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\.We define the function of similarity between𝐐t\(l\)\\mathbf\{Q\}\_\{t\}^\{\(l\)\}and𝐏t\(l\)​𝐖K\(l\)\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}as

Sim​\(𝐐t\(l\),𝐏t\(l\)​𝐖K\(l\)\)=exp​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\)\.\\text\{Sim\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\},\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)=\\text\{exp\}\\left\(\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\.Then, the memory attention can be rewritten as

𝐀ˇt\(l\)=diag​\(Sim​\(𝐐t\(l\),𝐏t\(l\)​𝐖K\(l\)\)⋅𝟏\)−1​Sim​\(𝐐t\(l\),𝐏t\(l\)​𝐖K\(l\)\)​\(𝐏t\(l\)​𝐖V\(l\)\)\.\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{diag\}\\left\(\\text\{Sim\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\},\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)\\cdot\\mathbf\{1\}\\right\)^\{\-1\}\\text\{Sim\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\},\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\.In order to decompose the𝐐t\(l\)\\mathbf\{Q\}\_\{t\}^\{\(l\)\}part and𝐏t\(l\)​𝐖K\(l\)\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}part, we approximate the similarity with

Sim​\(𝐐t\(l\),𝐏t\(l\)​𝐖K\(l\)\)=𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk\.\\text\{Sim\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\},\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)=\\frac\{\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\.
So the memory attention can be rewritten as

𝐀ˇt\(l\)=diag​\(𝐐t\(l\)​\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk⋅𝟏\)−1​𝐐t\(l\)​\[\(𝐏t\(l\)​𝐖K\(l\)\)⊤dk​\(𝐏t\(l\)​𝐖V\(l\)\)\]\.\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{diag\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\frac\{\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\cdot\\mathbf\{1\}\\right\)^\{\-1\}\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\[\\frac\{\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\left\(\\mathbf\{P\}^\{\(l\)\}\_\{t\}\\mathbf\{W\}\_\{V\}^\{\(l\)\}\\right\)\\right\]\.\(8\)Finally, we consider the prefix tokens𝐏t\(l\)\\mathbf\{P\}^\{\(l\)\}\_\{t\}as thecc\-th \(c<tc<t\) step aggregated results𝐇¯c\(l\)\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}, so we get

𝐀ˇt\(l\)=diag​\(𝐐t\(l\)​\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk⋅𝟏\)−1​𝐐t\(l\)​\[\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk​\(𝐇¯c\(l\)​𝐖~V\(l\)\)\],\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{diag\}\\left\(\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\cdot\\mathbf\{1\}\\right\)^\{\-1\}\\mathbf\{Q\}\_\{t\}^\{\(l\)\}\\left\[\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}\\right\)\\right\],\(9\)where𝐖~K\(l\),𝐖~V\(l\)\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\},\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}are parameters of the hyper memory block\. It is worth noting that, in practice, the reference stepccis not accessible in advance, and the evidence required at stepttmay span more than a single step\. Meanwhile, the memory key states and memory value states of different steps are coupled together within the fixed\-size memory network𝐌t\(l\)\\mathbf\{M\}^\{\(l\)\}\_\{t\}and the normalization vector𝐒t\(l\)\\mathbf\{S\}^\{\(l\)\}\_\{t\}, so that the non\-reference steps \(j≠cj\\neq c\) inevitably leak into the readout as noise\. To mitigate the impact of such noise, instead of directly reusing the vanilla attention query𝐐t\(l\)\\mathbf\{Q\}\_\{t\}^\{\(l\)\}inEquation \([9](https://arxiv.org/html/2607.26760#S3.E9)\), we utilize a trainable memory query𝐐~t\(l\)=𝐇~t\(l\)​𝐖~Q\(l\)\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}=\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}to get

𝐀ˇt\(l\)=diag​\(𝐐~t\(l\)​\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk⋅𝟏\)−1​𝐐~t\(l\)​\[\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk​\(𝐇¯c\(l\)​𝐖~V\(l\)\)\],\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}=\\text\{diag\}\\left\(\\tilde\{\\mathbf\{Q\}\}\_\{t\}^\{\(l\)\}\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\cdot\\mathbf\{1\}\\right\)^\{\-1\}\\tilde\{\\mathbf\{Q\}\}\_\{t\}^\{\(l\)\}\\left\[\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}\\right\)\\right\],\(10\)where𝐖~Q\(l\)∈ℝd×dk\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{k\}\}is an optimizable projection\. This decouples the memory query from the original attention and offers the freedom to reshape the cross\-step similarities𝐐~t\(l\)​𝐊~j\(l\)⊤\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}\. Therefore, it can emphasize the relevant steps while suppressing the irrelevant ones, thereby reducing the influence of noise when using the memory query states\. We provide a detailed theoretical error analysis inSection[3\.5](https://arxiv.org/html/2607.26760#S3.SS5)\.

### 3\.5Theoretical Error Analysis

Unlike standard Transformers that store all historical KV pairs in a growing cache, the hyper memory block compresses information into a fixed\-size matrix\. Suppose the model requires extracting information from thecc\-th step\. The dense memory network can be expressed as

𝐌t\(l\)=∑j=1t−1λt−\(j\+1\)⋅\(1−λ\)Lj′⋅𝐊~j\(l\)⊤dk​𝐕~j\(l\)\.\\mathbf\{M\}^\{\(l\)\}\_\{t\}=\\sum\_\{j=1\}^\{t\-1\}\\lambda^\{t\-\(j\+1\)\}\\cdot\\frac\{\(1\-\\lambda\)\}\{L\_\{j\}^\{\\prime\}\}\\cdot\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\tilde\{\\mathbf\{V\}\}\_\{j\}^\{\(l\)\}\.Then, we use the memory query𝐐~t\(l\)=𝐇~t\(l\)​𝐖~Q\(l\)\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}=\\tilde\{\\mathbf\{H\}\}^\{\(l\)\}\_\{t\}\\tilde\{\\mathbf\{W\}\}\_\{Q\}^\{\(l\)\}to extract information from the dense memory network and query\-key normalization vector by

𝐀~t\(l\)=\\displaystyle\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}=diag​\(𝐐~t\(l\)​𝐒t\(l\)\)−1​𝐐~t\(l\)​𝐌t\(l\)\\displaystyle\\text\{diag\}\\left\(\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\mathbf\{S\}^\{\(l\)\}\_\{t\}\\right\)^\{\-1\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\mathbf\{M\}^\{\(l\)\}\_\{t\}=\\displaystyle=diag​\(∑j=1t−1λt−\(j\+1\)​\(1−λ\)Lj′​𝐐~t\(l\)​𝐊~j\(l\)⊤dk​𝟏\)−1\\displaystyle\\text\{diag\}\\left\(\\sum\_\{j=1\}^\{t\-1\}\\lambda^\{t\-\(j\+1\)\}\\frac\{\(1\-\\lambda\)\}\{L\_\{j\}^\{\\prime\}\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\mathbf\{1\}\\right\)^\{\-1\}⋅\(∑j=1t−1λt−\(j\+1\)⋅\(1−λ\)Lj′​𝐐~t\(l\)​𝐊~j\(l\)⊤dk​𝐕~j\(l\)\)\.\\displaystyle\\cdot\\left\(\\sum\_\{j=1\}^\{t\-1\}\\lambda^\{t\-\(j\+1\)\}\\cdot\\frac\{\(1\-\\lambda\)\}\{L\_\{j\}^\{\\prime\}\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\tilde\{\\mathbf\{V\}\}\_\{j\}^\{\(l\)\}\\right\)\.We define the individual terms in the above summation as

𝐔j=λt−\(j\+1\)​\(1−λ\)Lj′​𝐐~t\(l\)​𝐊~j\(l\)⊤dk​𝟏,\\mathbf\{U\}\_\{j\}=\\lambda^\{t\-\(j\+1\)\}\\frac\{\(1\-\\lambda\)\}\{L\_\{j\}^\{\\prime\}\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\mathbf\{1\},𝐑j′=λt−\(j′\+1\)⋅\(1−λ\)Lj′′​𝐐~t\(l\)​𝐊~j′\(l\)⊤dk​𝐕~j′\(l\),\\mathbf\{R\}\_\{j^\{\\prime\}\}=\\lambda^\{t\-\(j^\{\\prime\}\+1\)\}\\cdot\\frac\{\(1\-\\lambda\)\}\{L\_\{j^\{\\prime\}\}^\{\\prime\}\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{j^\{\\prime\}\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\tilde\{\\mathbf\{V\}\}\_\{j^\{\\prime\}\}^\{\(l\)\},and𝐀~t\(l\)\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}can be rewritten as

𝐀~t\(l\)=\\displaystyle\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}=diag​\(∑j=1t−1𝐔j\)−1​\(∑j′=1t−1𝐑j′\)\.\\displaystyle\\text\{diag\}\\left\(\\sum\_\{j=1\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)^\{\-1\}\\left\(\\sum\_\{j^\{\\prime\}=1\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\\right\)\.Assume that our target information is stored at thecc\-step \(i\.e\.,the similarity𝐐~t\(l\)​𝐊~c\(l\)\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\tilde\{\\mathbf\{K\}\}\_\{c\}^\{\(l\)\}is significantly higher than others\)\. Then, we can further rewrite the equation as

𝐀~t\(l\)=\\displaystyle\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}=diag​\(𝐔c\+∑j=1,j≠ct−1𝐔j\)−1​\(𝐑c\+∑j′=1,j′≠ct−1𝐑j′\)\.\\displaystyle\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\+\\sum\_\{j=1,j\\neq c\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)^\{\-1\}\\left\(\\mathbf\{R\}\_\{c\}\+\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq c\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\\right\)\.=\\displaystyle=\(diag​\(𝐔c\)​\(𝐈\+diag​\(𝐔c\)−1⋅diag​\(∑j=1,j≠ct−1𝐔j\)\)\)−1​\(𝐑c\+∑j′=1,j′≠ct−1𝐑j′\)\.\\displaystyle\\left\(\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)\\left\(\\mathbf\{I\}\+\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\cdot\\text\{diag\}\\left\(\\sum\_\{j=1,j\\neq c\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)\\right\)\\right\)^\{\-1\}\\left\(\\mathbf\{R\}\_\{c\}\+\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq c\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\\right\)\.According to the first\-order Taylor expansion, we have

𝐀~t\(l\)≈\\displaystyle\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}\\approx\(𝐈−diag​\(𝐔c\)−1⋅diag​\(∑j=1,j≠ct−1𝐔j\)\)​diag​\(𝐔c\)−1​\(𝐑c\+∑j′=1,j′≠ct−1𝐑j′\),\\displaystyle\\left\(\\mathbf\{I\}\-\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\cdot\\text\{diag\}\\left\(\\sum\_\{j=1,j\\neq c\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)\\right\)\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\left\(\\mathbf\{R\}\_\{c\}\+\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq c\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\\right\),=\\displaystyle=diag​\(𝐔c\)−1​𝐑c\+diag​\(𝐔c\)−1​∑j′=1,j′≠ct−1𝐑j′⏟ϵ1−diag​\(𝐔c\)−1⋅diag​\(∑j=1,j≠ct−1𝐔j\)​diag​\(𝐔c\)−1​𝐑c⏟ϵ2\\displaystyle\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\mathbf\{R\}\_\{c\}\+\\underbrace\{\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq c\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\}\_\{\\boldsymbol\{\\epsilon\}\_\{1\}\}\-\\underbrace\{\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\cdot\\text\{diag\}\\left\(\\sum\_\{j=1,j\\neq c\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\mathbf\{R\}\_\{c\}\}\_\{\\boldsymbol\{\\epsilon\}\_\{2\}\}−\\displaystyle\-diag​\(𝐔c\)−1⋅diag​\(∑j=1,j≠ct−1𝐔j\)​diag​\(𝐔c\)−1​∑j′=1,j′≠ct−1𝐑j′⏟ϵ3,\\displaystyle\\underbrace\{\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\cdot\\text\{diag\}\\left\(\\sum\_\{j=1,j\\neq c\}^\{t\-1\}\\mathbf\{U\}\_\{j\}\\right\)\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq c\}^\{t\-1\}\\mathbf\{R\}\_\{j^\{\\prime\}\}\}\_\{\\boldsymbol\{\\epsilon\}\_\{3\}\},where the first term is equivalent to𝐀ˇt\(l\)\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}inEquation \([10](https://arxiv.org/html/2607.26760#S3.E10)\)with𝐊~c\(l\)=𝐇¯c\(l\)​𝐖~K\(l\)\\tilde\{\\mathbf\{K\}\}\_\{c\}^\{\(l\)\}=\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}and𝐕~c\(l\)=𝐇¯c\(l\)​𝐖~V\(l\)\\tilde\{\\mathbf\{V\}\}\_\{c\}^\{\(l\)\}=\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}by

diag​\(𝐔c\)−1​𝐑c=\\displaystyle\\text\{diag\}\\left\(\\mathbf\{U\}\_\{c\}\\right\)^\{\-1\}\\mathbf\{R\}\_\{c\}=diag​\(𝐐~t\(l\)​𝐊~c\(l\)⊤dk​𝟏\)−1​𝐐~t\(l\)​𝐊~c\(l\)⊤dk​𝐕~c\(l\)\\displaystyle\\text\{diag\}\\left\(\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{c\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\mathbf\{1\}\\right\)^\{\-1\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\tilde\{\\mathbf\{K\}\}\_\{c\}^\{\(l\)\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\tilde\{\\mathbf\{V\}\}\_\{c\}^\{\(l\)\}=\\displaystyle=diag​\(𝐐~t\(l\)​\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk​𝟏\)−1​𝐐~t\(l\)​\(𝐇¯c\(l\)​𝐖~K\(l\)\)⊤dk​\(𝐇¯c\(l\)​𝐖~V\(l\)\)\\displaystyle\\text\{diag\}\\left\(\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\mathbf\{1\}\\right\)^\{\-1\}\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\frac\{\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{K\}^\{\(l\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\left\(\\bar\{\\mathbf\{H\}\}^\{\(l\)\}\_\{c\}\\tilde\{\\mathbf\{W\}\}\_\{V\}^\{\(l\)\}\\right\)=\\displaystyle=𝐀ˇt\(l\)\.\\displaystyle\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}\.Therefore, there are three error terms for‖𝐀~t\(l\)−𝐀ˇt\(l\)‖2\|\|\\tilde\{\\mathbf\{A\}\}\_\{t\}^\{\(l\)\}\-\\check\{\\mathbf\{A\}\}^\{\(l\)\}\_\{t\}\|\|\_\{2\}\. It is worth noting thatϵ2\\boldsymbol\{\\epsilon\}\_\{2\}andϵ3\\boldsymbol\{\\epsilon\}\_\{3\}are structural errors caused by global normalization, whereasϵ1\\boldsymbol\{\\epsilon\}\_\{1\}is an attention error introduced by irrelevant information\. Across these three terms, there is always at least one factor of𝐐~t\(l\)​𝐊~j\(l\)⊤\\tilde\{\\mathbf\{Q\}\}^\{\(l\)\}\_\{t\}\\tilde\{\\mathbf\{K\}\}\_\{j\}^\{\(l\)\\top\}wherej≠cj\\neq cin the summation\. Therefore, when this similarity is low, the resulting error is expected to be small\.

### 3\.6Efficiency Analysis

Metis introduces native memory with limited additional inference overhead compared with external memory\. The key reason is that the original attention, memory attention, and memory storage procedure can be largely executed in parallel\. For thell\-th layer at steptt, the original attention branch computes token\-token attention over the current input, while the memory utilization branch performs memory attention over the native memory state𝐌t\(l\)\\mathbf\{M\}^\{\(l\)\}\_\{t\}and𝐒t\(l\)\\mathbf\{S\}^\{\(l\)\}\_\{t\}\. These two branches depend on the same input hidden states but have no sequential dependency on each other\. Therefore, memory attention does not need to wait for the output of the original attention, and its results can be fused only after both branches finish\.

The memory storage procedure can also be decoupled from the current inference path\. It updates the memory state for future steps, while the current step only reads from the existing memory state\. Thus, after the required hidden states are available, the storage branch can be executed in parallel with the original attention and memory utilization, instead of becoming an additional sequential stage\. As a result, the layer\-level latency can be expressed as

Tparallel\(l\)=max⁡\(Torig\(l\),Tutil\(l\),Tstore\(l\)\)\+Tfuse\(l\)\.T\_\{\\text\{parallel\}\}^\{\(l\)\}=\\max\\left\(T\_\{\\text\{orig\}\}^\{\(l\)\},T\_\{\\text\{util\}\}^\{\(l\)\},T\_\{\\text\{store\}\}^\{\(l\)\}\\right\)\+T\_\{\\text\{fuse\}\}^\{\(l\)\}\.\(11\)
Moreover, Metis stores historical information in fixed\-size native memory states, rather than appending retrieved textual memories to the input context\. Therefore, its memory utilization cost depends mainly on the memory state size, instead of growing linearly with the number of historical interactions\. This enables Metis to provide native memory capabilities while avoiding the retrieval, concatenation, and prefilling overhead commonly introduced by external memory systems\.

## 4Data Construction

In order to build Metis by mid\-training based on general foundation models, we synthesize a comprehensive training dataset based on existing public datasets\. This dataset consists of primary data and auxiliary data, which are used for training native memory procedures and improving generalization in complex scenarios\.

### 4\.1Primary Data

The primary data serves as the core supervision for training native memory procedures\. It is designed to teach memory foundation models to perform different memory operations in the forward computation through optimization, thereby generalizing to various scenarios\. The native memory procedure is acquired through optimization rather than manual rules, so the primary data must provide explicit supervision for the desired memory operations\.

Data Principles\.We highlight two data principles\. First, the data should be structured as a temporally ordered sequence of interaction steps, which mirrors the time\-streaming nature of online information\. Second, the data should be state\-consistent\. The response to a later query must agree with the memory state shaped by earlier operations\. Together, these two properties teach the model to store information and use it at the appropriate later step\.

Instead of generating data from scratch, we synthesize the primary data from established public benchmarks\. This choice offers three advantages\. Mature benchmarks provide verified facts and reasoning chains, which reduce hallucination when we extend them into long interaction sequences\. Their broad coverage of fiction, science, news, and logical reasoning enriches the context and improves generalization\. In addition, every synthetic sample is anchored to a source fact, which keeps the corpus traceable and easy to verify\.

Data Summary\.We select 27 public benchmarks across four memory operations, as shown inTable[1](https://arxiv.org/html/2607.26760#S4.T1)\. We organize the primary data along three orthogonal dimensions: \(1\)Memory operationincludesremember,forget,update, andreflect, which together span the core behaviors of native memory\. For every operation, a structured fact serves as the unit of synthesis, and the final query is answerable only from the information introduced in the preceding turns\. Aremembersample states a fact and then queries it, whereas areflectsample introduces several single\-hop facts and then queries their multi\-hop composition\. Anupdatesample modifies a previously stated fact before the query, and aforgetsample revokes a previously stated fact before the query\. \(2\)Salience of the instructionranges from explicit memory commands to implicit statements that embed information within natural narratives\. \(3\)Noise level, where clean sequences form the basic case and noisy sequences are produced by inserting irrelevant turns\. Jointly, these dimensions encourage the memory procedure to generalize across operations, instruction styles, and noise levels\.

Table 1:Summary of the primary data\. Samples for each memory operation are synthesized from public benchmarks and follow a distinct multi\-turn skeleton, where the answer to the final query stays consistent with the preceding memory operations\.OperationInteraction StreamingSource BenchmarksRememberInfo\(A1\\text\{A\}\_\{1\}\)→\\toQuery\(A\)LoCoMo\[maharana2024evaluating\], LongMemEval\[wu2024longmemeval\], NeedleInAHaystack\[kamradt2023needle\], RULER\[hsieh2024ruler\], LongBench\[bai2024longbench\],∞\\inftyBench\[zhang2024bench\], L\-Eval\[an2024eval\], BABILong\[kuratov2024babilong\], Bamboo\[dong2024bamboo\], NaturalQuestions\[kwiatkowski2019natural\], LongChat\-Eval\[li2023long\]UpdateInfo\(A1\\text\{A\}\_\{1\}\)→\\toInfo\(A2\\text\{A\}\_\{2\}\)→\\toQuery\(A\)ZsRE\[levy2017zero\], RippleEdits\[cohen2024evaluating\], KnowEdit\[zhang2024comprehensive\], TemporalWiki\[jang2022temporalwiki\]ForgetInfo\(A1\\text\{A\}\_\{1\}\)→\\toInfo\(A¯1\\bar\{\\text\{A\}\}\_\{1\}\)→\\toQuery\(A\)TOFU\[maini2024tofu\], WMDP\[li2024wmdp\], MUSE\[shi2025muse\], RWKU\[cao2024rwku\], WhoIsHarryPotter\[eldan2023s\], BLUR\[hu2026blur\], LKF\[singh2025unlearning\], CLEAR\[dontsov2025clear\], CounterFact\[meng2022locating\]ReflectInfo\(A1\\text\{A\}\_\{1\}\)→\\toInfo\(B1\\text\{B\}\_\{1\}\)→\\toQuery\(A∩B\\text\{A\}\\cap\\text\{B\}\)MuSiQue\[trivedi2022musique\], StrategyQA\[geva2021did\], Bamboogle\[press2023measuring\]

Construction Pipeline\.Our data synthesis pipeline comprises three major steps, including seed extraction, static synthesis, and quality verification\.

Step 1: Seed Extraction\.From each source dataset, we extract the source reference, query, and answer to form a base dialogue\. Then, we summarize the underlying fact into a structured seed, which records a subject, a relation, and a target, together with operation\-specific fields such as the updated target or the multi\-hop chain\. We also collect a pool of distractor dialogues that are logically orthogonal to each query, which are used to extend the sequence length\.

Step 2: Static Synthesis\.Guided by the structured seed, a strong instruction\-following language model rewrites each base dialogue into two salience styles\. The explicit style phrases the reference as a clear memory instruction, while the implicit style states the same fact as a description without an explicit instruction\. To cover long\-range memory, we insert a variable number of distractor turns between the reference and the query, which yields the distract variant of both styles\.

Step 3: Quality Verification\.A language model acts as an automatic judge and filters samples according to several quality criteria\. The consistency check confirms that the final answer faithfully reflects the intended memory state\. The orthogonality check ensures that inserted distractors do not leak the core fact, and the shortcut check removes any query that can be answered without its reference\. We additionally monitor the semantic diversity of the queries to prevent template collapse\. Samples that fail any check are discarded, so that only reliable samples enter the final corpus\.

Data Statistics\.We report the statistics of the synthesized primary data inTable[2](https://arxiv.org/html/2607.26760#S4.T2)\. After filtering, the corpus contains 357,137 samples and about 406 million tokens, drawn from 27 source benchmarks\. The samples are distributed across explicit, implicit, and distractor styles, which balance instruction salience and noise level\. The token count is dominated by the distractor samples, especially forremember, because long irrelevant contexts are inserted to strengthen long\-range memory\. This profile indicates that the primary data covers diverse memory operations at varied interaction lengths, which provides a solid basis for training native memory procedures\.

Table 2:Statistics of the synthesized primary data\. The explicit, implicit, and distract columns report the number of samples of each style, and the last column reports the total token count in millions\.OperationSourcesExplicitImplicitDistractAll SamplesTokens \(M\)Remember1114,68213,67128,50256,855362\.0Forget959,9008,25168,120136,27121\.7Update433,4527,30040,74981,50111\.0Reflect320,64620,61541,24982,51011\.4Total27128,68049,837178,620357,137406\.1
### 4\.2Auxiliary Data

The auxiliary data is used to improve the model’s generalizability\. It further enhances the capabilities of memory foundation models for complex scenarios, such as multi\-entity tasks and mixed dialogues\.

Construction Principles\.The primary data consists of basic memory operations interactions, ranging from single\-fact operations to multi\-fact reasoning cases\. However, real interactions are more complex\. Multiple similar facts may coexist, some facts may be revoked while others persist, and memory turns are often interleaved with ordinary conversation\.

The first is interference, where the model confuses similar facts or allows a forgetting operation to corrupt a retained fact in parametric spaces\. The second is memory pollution, where the model applies stored values to questions that do not need them\. The auxiliary data complements the primary data by targeting exactly these scenarios\. It preserves the same fact\-level structure, but composes facts and dialogues into more challenging interaction patterns, which improves the generalization and robustness of native memory\.

Data Summary\.We organize the auxiliary data into four subtypes, shown inTable[3](https://arxiv.org/html/2607.26760#S4.T3)\. The first two subtypes address multi\-fact scenarios\. \(1\)Multi\-entity bindingjointly stores two confusable facts and queries both, which trains the model to bind each value to its own fact\. \(2\)Selective forgettingrevokes one fact while the other persists, which trains the model to forget one fact selectively without collateral loss\. The other two subtypes address memory pollution\. \(3\)Post\-memory dialoguecontinues an ordinary conversation right after a memory query, so the model does not carry stored values into unrelated answers\. \(4\)Memory\-irrelevant dialogueanswers a question that does not need memory even when a memory state exists, so the model learns when memory should not influence the response\. Therefore, these four subtypes extend the primary data to realistic mixed interactions\.

Table 3:Summary of the auxiliary data\. Each subtype composes facts or dialogues into a complex interaction pattern, where the final answers remain consistent with the intended memory state\.Auxiliary SubtypeInteraction StreamingConstruction SourceMulti\-Entity BindingInfo\(A1\\text\{A\}\_\{1\}\)→\\toInfo\(B1\\text\{B\}\_\{1\}\)→\\toQuery\(A\)→\\toQuery\(B\)Paired facts synthesized from primary source factsSelective ForgettingInfo\(A1\\text\{A\}\_\{1\}\)→\\toInfo\(B1\\text\{B\}\_\{1\}\)→\\toInfo\(B¯1\\bar\{\\text\{B\}\}\_\{1\}\)→\\toQuery\(B\)→\\toQuery\(A\)Paired facts synthesized from primary source factsPost\-Memory DialogueInfo\(A1\\text\{A\}\_\{1\}\)→\\toQuery\(A\)→\\toChatPrimary memory samples with curated normal dialoguesMemory\-Irrelevant DialogueInfo\(A1\\text\{A\}\_\{1\}\)→\\toChat / Chat→\\toChatPrimary memory samples with curated normal dialoguesConstruction Pipeline\.The auxiliary data is built from two shared ingredients, including synthesized paired facts and prepared normal dialogues, which are then formulated into the four subtypes\.

Paired Fact Synthesis\.The multi\-entity binding and selective forgetting subtypes require pairs of similar facts\. For each source fact, represented by a subject, relation, and value, we synthesize one confusable counterpart fact\. Each counterpart is generated using one of four transformations relative to the source fact\. It keeps the subject but changes the relation, keeps the relation but changes the subject, imitates the value format, or stays semantically adjacent\. A language model generates each counterpart, and a verifier discards any fact that contradicts, restates, or depends on the source\. We then rewrite the verified facts into natural statements, queries, and revocation snippets to ensure fluency and diversity\.

Normal Dialogue Preparation\.The post\-memory dialogue and memory\-irrelevant dialogue subtypes require conversations that do not require access to memory\. We prepare a dialogue pool from three sources\. These are general assistant dialogues for everyday requests, open\-domain conversations from public corpora, and entity\-related dialogues that are topically related to a stored fact yet remain answerable without it\. We filter out turns with memory cues, real\-time facts, or unsafe content, and we remove duplicates\.

Subtype Formulation\.The multi\-entity binding subtype states the paired facts in turn and then queries both, which forces the model to bind each value to its correct fact\. The selective forgetting subtype states both facts, revokes one, and then queries both, so the revoked fact becomes unavailable while the retained fact stays correct\. The post\-memory dialogue subtype appends an unrelated ordinary turn after a memory query, so the model returns to normal conversation without leaking any stored value\. The memory\-irrelevant dialogue subtype keeps the memory state but drops its query before an ordinary turn, and it also includes standalone dialogues that carry no memory at all\.

Data Statistics\.We report the statistics of the synthesized auxiliary data inTable[4](https://arxiv.org/html/2607.26760#S4.T4)\. The paired\-fact synthesis yields 76,153 natural snippet sets after quality verification\. These snippets support 76,153 multi\-entity binding samples and 76,153 selective forgetting samples\. The dialogue\-based subtypes are larger, because they reuse the full set of primary memory samples\. Post\-memory dialogue contributes 357,137 samples, and memory\-irrelevant dialogue contributes 100,000 samples\. In total, the auxiliary data adds 609,443 samples that emphasize multi\-fact reasoning and pollution\-resistant conversation\. Together with the primary data, it provides broad coverage from single\-fact operations to complex mixed interactions\.

Table 4:Statistics of the synthesized auxiliary data\. We report the number of samples for each subtype, and we exclude every sample that fails quality verification\.TargetAuxiliary SubtypeSamplesMulti\-fact ScenarioMulti\-Entity Binding76,153Selective Forgetting76,153Memory PollutionPost\-Memory Dialogue357,137Memory\-Irrelevant Dialogue100,000AllTotal609,443

## 5Model Optimization

To empower Metis with native memory procedures, we design multiple training objectives for mid\-training\. These objectives primarily consist of memory reconstruction, memory operation, and regularization\. The three objectives share a common likelihood form but operate on different data\. They jointly shape the native memory state and procedure\.

### 5\.1Overview

We organize every training sample as a multi\-step interactions=\{\(Xt,Yt\)\}t=1Tss=\\\{\(X\_\{t\},Y\_\{t\}\)\\\}\_\{t=1\}^\{T\_\{s\}\}, following the definition inSection[2](https://arxiv.org/html/2607.26760#S2)\. At steptt, the model reads the input instructionXtX\_\{t\}and generates the assistant responseYt=\(yt,1,…,yt,\|Yt\|\)Y\_\{t\}=\(y\_\{t,1\},\\dots,y\_\{t,\|Y\_\{t\}\|\}\)\. All steps are forwarded sequentially, and the native memory procedure stores information from each step to the memory state before the next step starts\. Therefore, the parametersθt\\theta\_\{t\}at stepttalready integrate the memory state from all preceding steps\{\(Xi,Yi\)\}i<t\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i<t\}\. We supervise only a subset of query steps𝒬s⊆\{1,…,Ts\}\\mathcal\{Q\}\_\{s\}\\subseteq\\\{1,\\dots,T\_\{s\}\\\}, where the assistant response is labeled, while the reference and operation steps remain unlabeled\. However, the responses of the reference and operation steps are still generated or provided for memory state updates\. In addition, all three objectives share the per\-step loss below, and they differ only in how the supervised targetYtY\_\{t\}is constructed from the training data\. For a supervised stept∈𝒬st\\in\\mathcal\{Q\}\_\{s\}, we define the per\-step loss as the token\-averaged negative log\-likelihood

ℓ​\(s,t\)=−1\|Yt\|​∑k=1\|Yt\|log⁡P​\(yt,k∣Xt,Yt,<k;θt\),\\ell\(s,t\)=\-\\frac\{1\}\{\|Y\_\{t\}\|\}\\sum\_\{k=1\}^\{\|Y\_\{t\}\|\}\\log P\(y\_\{t,k\}\\mid X\_\{t\},Y\_\{t,<k\};\\theta\_\{t\}\),\(12\)whereYt,<kY\_\{t,<k\}denotes the previously generated tokens at steptt, andθt\\theta\_\{t\}is conditioned on the memory state shaped by earlier steps\. The loss of a sample aggregates over its supervised steps as∑t∈𝒬sℓ​\(s,t\)\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\), which allows a single trajectory to supervise multiple responses\. During mid\-training, we freeze the backbone parameters and optimize only the native memory parameters\.

The three objectives correspond to five data subsets, and we control their contributions through a task\-weighted sampler rather than explicit loss coefficients\. At training epochee, the sampling probability of subsetτ\\tauis

πτ​\(e\)=wτ​\(e\)∑τ′∈𝒯wτ′​\(e\),wτ​\(e\)=wτs\+\(wτe−wτs\)⋅min⁡\(eE−1,1\),\\pi\_\{\\tau\}\(e\)=\\frac\{w\_\{\\tau\}\(e\)\}\{\\sum\_\{\\tau^\{\\prime\}\\in\\mathcal\{T\}\}w\_\{\\tau^\{\\prime\}\}\(e\)\},\\qquad w\_\{\\tau\}\(e\)=w\_\{\\tau\}^\{\\text\{s\}\}\+\\left\(w\_\{\\tau\}^\{\\text\{e\}\}\-w\_\{\\tau\}^\{\\text\{s\}\}\\right\)\\cdot\\min\\\!\\left\(\\frac\{e\}\{E\-1\},1\\right\),\(13\)where𝒯\\mathcal\{T\}is the set of subsets,EEis the total number of epochs, andwτs,wτew\_\{\\tau\}^\{\\text\{s\}\},w\_\{\\tau\}^\{\\text\{e\}\}are the start and end weights of subsetτ\\tau\. This linear annealing forms a curriculum that gradually shifts the sampling mass from storage\-oriented data toward harder long\-range and regularization data\. Because the weights only modulate the sampling frequency, the expected mid\-training objective can be written as

ℒ=∑τ∈𝒯πτ​\(e\)⋅𝔼s∼𝒟τ​\[∑t∈𝒬sℓ​\(s,t\)\],\\mathcal\{L\}=\\sum\_\{\\tau\\in\\mathcal\{T\}\}\\pi\_\{\\tau\}\(e\)\\cdot\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\tau\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\],\(14\)where𝒟τ\\mathcal\{D\}\_\{\\tau\}is the data of subsetτ\\tau, and every sampled step contributes an unweighted loss fromEquation \([12](https://arxiv.org/html/2607.26760#S5.E12)\)\.

### 5\.2Memory Reconstruction Objective

The memory reconstruction objective enables Metis to store and reconstruct information\. It provides an important training signal during the model’s warm\-up phase, as initialized models typically lack such capabilities\. Furthermore, it targets the upper bound of information storage, with completely lossless compression and reconstruction\. However, a trade\-off exists between this objective and the native memory procedure\. First, reconstruction and instruction following are contradictory, as they require specificity and generalization, respectively\. Second, from the perspective of prediction tasks, the native memory procedure requires lossy compression guided by input instructions\. In contrast, memory reconstruction opposes lossy compression\.

This objective is built on a reconstruction subset derived from the primary data inSection[4\.1](https://arxiv.org/html/2607.26760#S4.SS1), denoted as𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}\. In each sample, a reference passage is presented and stored into the memory state at an early step, and a later query step requires the model to regenerate its content\. Because the supervised responseYtY\_\{t\}reproduces the stored reference, the memory state must retain the source with minimal loss\. We instantiate the per\-step loss over this subset as

ℒrec=πrec​\(e\)​𝔼s∼𝒟rec​\[∑t∈𝒬sℓ​\(s,t\)\],\\mathcal\{L\}\_\{\\text\{rec\}\}=\\pi\_\{\\text\{rec\}\}\(e\)\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\text\{rec\}\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\],\(15\)where the expectation averages over samples drawn from𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}, andℓ​\(s,t\)\\ell\(s,t\)measures the negative log\-likelihood of reconstructing the stored content at the query steptt\. By steptt, the reference passage has already been stored in the native memory state represented withinθt\\theta\_\{t\}\. Minimizingℒrec\\mathcal\{L\}\_\{\\text\{rec\}\}thus drives the hyper memory block to encode the reference into a state from which the memory utilization procedure can recover it\.

### 5\.3Memory Operation Objective

While reconstruction establishes lossless storage, native memory must additionally support input\-driven operations\. The memory operation objective teaches Metis to remember, forget, update, and reflect, so that the memory state evolves according to the instruction at each step\. It is built on the primary data, which exhibits these operations under controlled instruction salience and noise\.

We use two complementary subsets of the primary data\. The first subset, denoted as𝒟ope/i\\mathcal\{D\}\_\{\\text\{op\}\}^\{\\text\{e/i\}\}, contains the explicit and implicit samples\. Explicit samples phrase the operation as a clear command, whereas implicit samples embed the same information within a natural narrative\. This contrast forces the model to infer the operation from intent rather than from surface keywords\. The second subset, denoted as𝒟opd\\mathcal\{D\}\_\{\\text\{op\}\}^\{\\text\{d\}\}, contains the distractor samples, where irrelevant turns are inserted between the reference and the query\. It promotes long\-range retention and robustness against intervening noise\.

In all operation samples, the supervised response stays consistent with the information from the earlier steps\. For anupdatesample, the answer reflects the new value rather than the old one\. For aforgetsample, the answer no longer exposes the forgotten value\. For areflectsample, the answer composes several stored facts into multi\-hop reasoning\. Therefore, a single likelihood objective suffices to supervise all operations as

ℒop=πe/i​\(e\)​𝔼s∼𝒟ope/i​\[∑t∈𝒬sℓ​\(s,t\)\]\+πd​\(e\)​𝔼s∼𝒟opd​\[∑t∈𝒬sℓ​\(s,t\)\],\\mathcal\{L\}\_\{\\text\{op\}\}=\\pi\_\{\\text\{e/i\}\}\(e\)\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\text\{op\}\}^\{\\text\{e/i\}\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\]\+\\pi\_\{\\text\{d\}\}\(e\)\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\text\{op\}\}^\{\\text\{d\}\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\],\(16\)whereθt\\theta\_\{t\}now encodes the net effect of the preceding operation sequence on the memory state\. Unlike reconstruction, the target is no longer a copy of the stored content, so the model learns to transform and read the memory state under the guidance of the instruction\.

### 5\.4Regularization Objective

The reconstruction and operation objectives are primarily built on simple interaction patterns, which leave the model vulnerable in complex scenarios\. The regularization objective mitigates two failure modes that arise when memory operates in realistic interactions\. The first is interference, where similar facts are confused or a forgetting operation corrupts a retained fact\. The second is memory pollution, where stored values leak into responses that do not require them\[lin2026surveylongtermmemorysecurity\]\. This objective is built on the auxiliary data inSection[4\.2](https://arxiv.org/html/2607.26760#S4.SS2), which composes facts and dialogues into more complex and realistic interaction patterns\.

We use two subsets of the auxiliary data\. The multi\-fact subset𝒟mf\\mathcal\{D\}\_\{\\text\{mf\}\}targets interference\. Its multi\-entity binding samples jointly present two confusable facts and query the model about both, which constrains the memory utilization procedure to bind each value to its own key\. Its selective forgetting samples include an instruction that revokes one fact while preserving the other, which constrains the forget operation to act locally\. The memory pollution subset𝒟mp\\mathcal\{D\}\_\{\\text\{mp\}\}targets leakage\. Its post\-memory dialogue samples continue an ordinary conversation right after a memory query, and its memory\-irrelevant samples answer a question that needs no memory even when a memory state exists\. In both cases, they discourage the model from injecting memory into unrelated responses\.

These subsets act as regularization because they constrain memory behavior under more realistic and diverse interaction scenarios\. The supervised targets penalize cross\-fact interference, collateral forgetting, and value leakage, which suppress degenerate solutions that always read or overwrite the memory state\. We define the objective as

ℒreg=πmf​\(e\)​𝔼s∼𝒟mf​\[∑t∈𝒬sℓ​\(s,t\)\]\+πmp​\(e\)​𝔼s∼𝒟mp​\[∑t∈𝒬sℓ​\(s,t\)\],\\mathcal\{L\}\_\{\\text\{reg\}\}=\\pi\_\{\\text\{mf\}\}\(e\)\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\text\{mf\}\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\]\+\\pi\_\{\\text\{mp\}\}\(e\)\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\_\{\\text\{mp\}\}\}\\\!\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\_\{s\}\}\\ell\(s,t\)\\right\],\(17\)where many samples expose multiple supervised steps, so\|𝒬s\|\>1\|\\mathcal\{Q\}\_\{s\}\|\>1jointly constrains the retained and the revoked facts within one interaction\. For the memory\-irrelevant case, the supervised step is an ordinary turn whose target is independent of the memory state\.

## 6Experiments

### 6\.1Experimental Settings

We evaluate Metis on memory operation tasks and memory\-based question\-answering \(QA\) tasks\. The memory operation task evaluates the performance of executing memory operations\. In addition, to verify the effectiveness of the native memory state, we evaluate the performance on the memory\-based QA task\. Our major experiments focus on evaluating the native memory state and procedure primarily through relatively short\-term tasks\. As for the long\-term capability, we explore it from the perspective of memory capability inSection[6\.5](https://arxiv.org/html/2607.26760#S6.SS5)\.

Datasets and Metrics\.For memory operations, we employ MemOps\[hao2026memopsbenchmarkinglifecyclememory\], which is a specific benchmark focusing on memory operations, such as remembering, forgetting, and updating\. In the*Full*setting, the model receives three complete evidence segments, containing 24 utterances\. In the*Gold*setting, it receives only the oracle turns required for the question\. We also present the performance of theTestset of our constructed dataset\. For the memory\-based QA task, we conduct experiments on the golden\-session setting of LoCoMo \(i\.e\.,LoCoMo \(Gold\)\), where we provide the gold evidence sessions as input\. We also utilize the contextual generation task dataset from NextMem\[zhang2026nextmem\]for further analysis\. This dataset evaluates whether models can utilize the provided information to answer questions correctly, consisting of SQuAD\[rajpurkar2016squad\], HotpotQA\[yang2018hotpotqa\], LoCoMo\[maharana2024evaluating\], and LongMemEval\[wu2024longmemeval\]\. In all these settings, we utilize gpt\-4\.1\-mini to judge each prediction against its reference answer in three repeated evaluations\. Then, we report the median LLM\-as\-a\-judge score\. In each dataset, we calculate the average performance \(i\.e\.,Avg\.\) across different types using a micro\-average\. It should be noted that, to cover a wide range of entities for memory, we extract seed entities from various public datasets to synthesize our training data, such as LoCoMo and LongMemEval\. However, we do not leak their exact QA behaviors in the training phase\.

Baselines\.We comprehensively evaluate our approach against four categories of baselines\. For backbone models evaluated with full information appended to the context, we utilize Qwen3\.5\[qwen3\_5\]across 4B, 9B, and 27B sizes\. For the partial\-context baselines, we apply RAG\[lewis2020retrieval\]to these backbone models\. It encodes observations and queries into dense representations, and calculates the cosine similarity between queries and all observations\. The top\-55observations are appended to the context\. For TTT\-based models, we evaluate Temp\-LoRA\[wang2024greater\]as the baseline\. It fuses information into the model by training a temporary LoRA module on previous text chunks during inference, encoding historical context as transient parameter updates\. Specifically, we implement Temp\-LoRA with corresponding sizes of Qwen3\.5 backbones\. Regarding parametric memory models, we compare withδ\\delta\-Mem\[lei2026delta\], which steers attention with low\-rank corrections\. More details are provided in Appendix[E](https://arxiv.org/html/2607.26760#A5)\.

Training Configuration\.The reported Metis models are built upon Qwen3\.5 backbones and trained on8×8\\timesH100 GPUs\. The backbone is frozen during training, and the trainable memory parameters are initialized using the key and value projection matrices of the corresponding backbone layers\. We use AdamW with a learning rate of2×10−42\\times 10^\{\-4\}, a constant schedule after 200 warmup steps, weight decay 0\.01,β=\(0\.9,0\.999\)\\beta=\(0\.9,0\.999\),ϵ=10−8\\epsilon=10^\{\-8\}, and gradient clipping at 1\.0\. Training uses BF16 and seed 42, and saves a checkpoint every 2,000 steps\. For Metis\-4B, we train our model for 14,000 steps, corresponding to one epoch\. For Metis\-27B, we use the same number of training steps, corresponding to approximately 0\.4 epochs\. For Metis\-9B, we use 8,000 steps \(approximately 0\.5728 epochs\), which is selected by early stopping on validation\-set performance\.

Evaluation Pipeline\.For the memory operation and memory\-based QA tasks, we adopt a static evaluation paradigm\. Each test trajectory is divided into two sequential phases\. The first phase consists of information steps, which provide the necessary context to the model\. The second phase consists of query steps, where the model must answer a question based on the prior information\. Finally, the evaluation calculates performance metrics by comparing the model’s output in the query step with the ground truth\. The prompts of the information step and query step are provided inAppendix[F\.2](https://arxiv.org/html/2607.26760#A6.SS2), and the prompts of LLM\-as\-a\-Judge are presented inAppendix[F\.3](https://arxiv.org/html/2607.26760#A6.SS3)\.

### 6\.2Overall Performance

Table 5:The overall performance on memory operation tasks\. Full\-context and partial\-context results are shown in gray to visually distinguish context\-access settings from the no\-context comparison\. Within the No Context setting, the best and second\-best scores areboldedandunderlined, respectively\. Avg\. represents the micro\-average performance\.TypeMethodMemOps \(Gold\)Metis Test SetRememberUpdateForgetReflectAvg\.RememberUpdateForgetReflectAvg\.Full ContextQwen3\.5\-4B84\.9786\.3481\.3685\.1784\.5680\.0770\.3170\.4283\.1375\.18Qwen3\.5\-9B88\.5488\.4382\.7386\.9086\.8678\.1869\.4867\.5084\.3873\.89Qwen3\.5\-27B91\.3790\.7484\.3284\.4887\.9081\.0173\.4475\.8388\.7578\.87Partial ContextQwen3\.5\-4B38\.8433\.5624\.5521\.9030\.1870\.0563\.1259\.9067\.8164\.82Qwen3\.5\-9B30\.5126\.6220\.2311\.5522\.4170\.4055\.1055\.0064\.6960\.68Qwen3\.5\-27B37\.0535\.8822\.0516\.2128\.0170\.2863\.3366\.4660\.4765\.40No ContextQwen3\.5\-4B4\.170\.001\.590\.001\.6512\.030\.0049\.580\.0016\.96Qwen3\.5\-9B4\.171\.850\.910\.001\.8811\.795\.4249\.900\.6318\.64Qwen3\.5\-27B3\.570\.930\.910\.691\.6910\.732\.0847\.501\.2516\.87Temp\-LoRA\-4B15\.3310\.192\.954\.838\.8515\.8027\.7115\.2117\.6619\.34Temp\-LoRA\-9B23\.8113\.435\.008\.1013\.5120\.0517\.7120\.2120\.4719\.51Temp\-LoRA\-27B20\.686\.482\.504\.839\.7025\.9418\.6525\.2126\.8723\.86δ\\delta\-Mem7\.446\.021\.821\.554\.3813\.9221\.7712\.4010\.3115\.03Metis\-4B19\.3527\.557\.2716\.9017\.8452\.2463\.8531\.2590\.1656\.72Metis\-9B25\.8923\.6111\.5915\.5219\.6358\.1463\.3330\.4290\.7857\.92Metis\-27B28\.2731\.0210\.9126\.5524\.7661\.0868\.1377\.5093\.4473\.77

Memory Operation Tasks\.The results of MemOps in the gold setting \(i\.e\.,MemOps \(Gold\)\) and the Metis test set are presented inTable[5](https://arxiv.org/html/2607.26760#S6.T5)\. Due to the page limitation, we put the experiment results and analysis of MemOps in the full setting \(i\.e\.,MemOps \(Full\)\) inAppendix[B](https://arxiv.org/html/2607.26760#A2)\. As expected, full\-context models achieve the strongest overall performance, while removing the context causes a substantial performance drop for standard backbones\. Partial context preserves some information on the Metis test set but performs poorly on MemOps \(Gold\), showing that incomplete histories cannot reliably support memory operations\. Temp\-LoRA andδ\\delta\-Mem recover part of the lost performance, but their gains remain limited\. Under the same no\-context setting, Metis achieves the best average results on both MemOps \(Gold\) and the Metis test set\. These results suggest that Metis can preserve information in its native memory state and use it in later steps without replaying the original context\.

Metis\-27B achieves the best average performance on both benchmarks under the no\-context setting\. Compared with Metis\-4B and Metis\-9B, it shows clear gains in remembering, updating, reflection, and overall performance\. The improvement is especially large for forgetting on the Metis test set\. These results suggest that a sufficiently large backbone can better formulate and utilize the native memory state\. In addition, forgetting is still the most difficult operation on the external MemOps \(Gold\) benchmark, even for Metis\-27B\. This suggests that removing or suppressing information in a shared latent state is more difficult to generalize than storing or updating information\. Overall, Metis shows strong performance on memory operation tasks in short\-term scenarios\.

Table 6:The overall performance on memory\-based QA tasks\. Full\-context and partial\-context results are shown in gray to visually distinguish context\-access settings from the no\-context comparison\. Within the No Context setting, the best and second\-best scores areboldedandunderlined, respectively\. Single and Multi indicate Single\-hop Retrieval and Multi\-hop Retrieval settings in LoCoMo \(Gold\), respectively\. Temporal and Open refer to the temporal reasoning setting and open domain knowledge settings in LoCoMo \(Gold\), respectively\. A dash indicates that the corresponding result is not applicable\. Avg\. represents the micro\-average performance\.TypeMethodLoCoMo \(Gold\)NextMem \(Contextual Generation\)SingleMultiTemporalOpenAvg\.SQuADHotpotQALongMemEvalLoCoMoAvg\.Full ContextQwen3\.5\-4B85\.1265\.9215\.7823\.8863\.5291\.0088\.2845\.2161\.0077\.15Qwen3\.5\-9B84\.4364\.9316\.6421\.3563\.0091\.0087\.6145\.7960\.6577\.05Qwen3\.5\-27B85\.8369\.9615\.5531\.1865\.0391\.8089\.1848\.4364\.4778\.80Partial ContextQwen3\.5\-4B36\.2810\.796\.175\.6223\.54\-\-\-\-\-Qwen3\.5\-9B34\.058\.277\.502\.8121\.97\-\-\-\-\-Qwen3\.5\-27B35\.6310\.077\.731\.9723\.17\-\-\-\-\-No ContextQwen3\.5\-4B0\.000\.360\.001\.970\.1811\.2425\.222\.860\.4811\.86Qwen3\.5\-9B0\.000\.360\.000\.000\.0714\.9234\.982\.860\.4815\.93Qwen3\.5\-27B0\.000\.360\.000\.000\.0716\.7338\.903\.140\.4817\.75Temp\-LoRA\-4B10\.9211\.241\.8026\.699\.9926\.1938\.629\.7111\.1224\.20Temp\-LoRA\-9B13\.3313\.312\.4225\.0011\.7229\.5245\.0712\.9312\.8028\.12Temp\-LoRA\-27B4\.295\.491\.2510\.674\.2437\.6851\.466\.575\.8630\.97δ\\delta\-Mem12\.8610\.163\.2820\.2210\.7920\.7433\.029\.7910\.2920\.42Metis\-4B18\.9015\.297\.0328\.3716\.3129\.6258\.1339\.3650\.4841\.69Metis\-9B18\.8718\.537\.0327\.2516\.8133\.0663\.4533\.3651\.5643\.39Metis\-27B31\.0127\.9713\.8328\.9326\.7443\.4266\.5439\.7160\.4150\.82

Memory\-based QA Tasks\.The results of the memory\-based QA tasks are presented inTable[6](https://arxiv.org/html/2607.26760#S6.T6)\. Full\-context models provide a strong upper bound because they can directly attend to the original evidence\. Their performance drops sharply when only partial context is available\. Without context, the original Qwen3\.5 models obtain almost zero scores on LoCoMo \(Gold\), confirming that the answers cannot be reliably recovered from backbone knowledge alone\. In contrast, Metis achieves the best average performance on both benchmarks under the no\-context setting\. Metis\-27B obtains the highest score in every task category, outperforming other baselines\. These results show that the native memory state can preserve useful information and support question answering without replaying the original context\.

The advantage of Metis is especially clear on tasks with complex or long\-range memory requirements\. On NextMem \(Contextual Generation\), Metis achieves large gains on HotpotQA, LongMemEval, and LoCoMo subset\. It also substantially improves multi\-hop and temporal question answering on LoCoMo \(Gold\)\. It indicates that native memory remains effective for relatively simple factual questions while providing larger gains on more demanding tasks\. The strong improvement on temporal questions also suggests that a larger Metis model can better preserve and use relations across different interaction steps\. However, the gain on open\-domain LoCoMo \(Gold\) questions is relatively limited, which indicates that some task types remain difficult even with increased model capacity\.

In addition, we find that the improvement between Metis\-4B and Metis\-9B is modest, whereas Metis\-27B substantially improves the average score\. This pattern suggests that backbone scaling can enhance native memory capability once model capacity is reached, although the gains are not uniform across tasks\. These results indicate that Metis provides strong memory\-based QA performance in both the short\-term QA tasks and the relatively longer LoCoMo \(Gold\) setting\.

### 6\.3Ablation Studies

We conduct ablation studies on Metis\-4B from the perspectives of training data and model structure\. Following the main experimental setup, we evaluate LoCoMo \(Gold\) and NextMem in memory\-based QA tasks\. We also use the Metis test set and MemOps \(Gold\) in memory operation tasks\. All ablation models use Metis\-4B and the same training configuration and evaluation pipeline as the main results\.

Data Ablation\.We evaluate the contribution of different training data through two variants\. Inw/o MS, we remove the Multi\-fact Scenario data\. Inw/o MS\+MP, we remove the entire auxiliary dataset, including Multi\-fact and Memory Pollution data, to examine its overall contribution to memory learning and generalization\.

As shown inTable[7](https://arxiv.org/html/2607.26760#S6.T7), removing the Multi\-fact Scenario data consistently reduces performance across both types of tasks\. This result indicates that multi\-fact supervision helps Metis integrate related information and maintain a coherent memory state\. The decline is more evident on MemOps \(Gold\) and the Metis test set, suggesting that such data is particularly important for learning reliable memory operations\. Removing the entire auxiliary dataset leads to a much larger overall degradation\. The drop is especially clear on the Metis test set, while performance on LoCoMo \(Gold\) and NextMem also decreases consistently\. This shows that auxiliary data improves the robustness and generalization of native memory procedures across different scenarios\.

Structure Ablation\.We further ablate the main components of the native memory procedure\. Inw/o GDU, we replace the GDU with a linear update \(LU\)\. Inw/o SA, we remove the adaptive aggregation mechanism and directly use the last\-token hidden state for memory storage\. Inw/o OQ, we remove the optimizable memory query projection and reuse the query from the original attention\. Inw/o QKN, we remove query\-key normalization from memory attention\.

Among all these evaluated variants, removing adaptive aggregation causes the largest performance drop\. Directly using the last token cannot effectively capture information distributed across the input sequence\. As a result, the model fails to construct an informative memory state\. In addition, removing query\-key normalization also causes a substantial degradation, particularly on LoCoMo \(Gold\) and NextMem\. Without this normalization, irrelevant information may introduce stronger interference\. Furthermore, reusing the original attention query also reduces performance across all benchmarks\. The decrease is larger on the memory\-based QA tasks, indicating that a separate memory query is important for distinguishing relevant historical information from noise\. This observation is consistent with our theoretical analysis, where the additional query projection reshapes cross\-step similarities and suppresses interference from irrelevant memory\.

We also find that replacing the GDU with a linear update has a small effect on the overall average\. The linear update performs slightly better on MemOps \(Gold\) and the Metis test set but is clearly weaker on LoCoMo \(Gold\)\. This suggests that a linear update can handle simple and short\-term memory operations, while the GDU provides a better balance in long\-term scenarios\. This result is consistent with our engineering observation that GDU may produce more stable model behavior, motivating its use in Metis\.Appendix[C](https://arxiv.org/html/2607.26760#A3)further compares the per\-task performance of LU and GDU at the 9B and 27B scales\.

Table 7:Results of the ablation studies on Metis\-4B\. We use Avg\. to report the macro\-average performance in different groups, and utilizeΔ\\DeltaAvg\. to represent the relative performance gap compared with the full model\.TypeModelMemory Operation TaskMemory\-based QA TaskOverallMemOps \(Gold\)Metis Test SetAvg\.𝚫\\DeltaAvg\.LoCoMo \(Gold\)NextMemAvg\.𝚫\\DeltaAvg\.Avg\.𝚫\\DeltaAvg\.Full ModelMetis17\.8456\.7237\.28\-16\.3141\.6929\.00\-33\.14\-DataAblationw/o MS14\.6451\.5333\.08\-11\.26%14\.7837\.7926\.29\-9\.36%29\.68\-10\.43%w/o MS\+MP14\.4542\.1728\.31\-24\.07%14\.1836\.1725\.17\-13\.20%26\.74\-19\.31%StructureAblationw/o GDU18\.5058\.5438\.523\.32%11\.9742\.7827\.37\-5\.60%32\.95\-0\.58%w/o SA3\.6719\.7211\.70\-68\.63%9\.8418\.4914\.16\-51\.16%12\.93\-60\.98%w/o OQ13\.8953\.9333\.91\-9\.04%11\.4637\.0724\.26\-16\.33%29\.09\-12\.23%w/o QKN9\.3248\.7429\.03\-22\.13%10\.0026\.8018\.40\-36\.55%23\.72\-28\.44%

### 6\.4Out\-of\-Distribution Memory Tasks

To examine whether the strong performance reported inSection[6\.2](https://arxiv.org/html/2607.26760#S6.SS2)generalizes beyond the data\-construction distribution, we further evaluate Metis on two out\-of\-distribution \(OOD\) benchmarks that were not used to construct the training data\. All methods are evaluated under the no\-context setting\.

We evaluate ATM\-Bench\[mei2026atm\]on its official standard split\. For MemDaily\[zhang2026memsim\], we use the subset of the official pre\-generated release in which the annotated retrieval\-target messages occur before the query\. In the Gold setting, the model receives only benchmark\-annotated evidence: human\-annotated memory items represented as text in SGM for ATM\-Bench, and retrieval\-target messages for MemDaily\. ATM\-Bench scores list\-recall, number, and open\-ended questions using Jaccard similarity, post\-processed exact match, and an LLM judge, respectively\. MemDaily reports deterministic single\-choice accuracy for all six question types\.

Table 8:Results on OOD memory benchmarks\. The best and unique second\-best scores areboldedandunderlined, respectively\. The average score is calculated according to the official category counts\.MethodATM\-Bench \(Gold\)MemDaily \(Gold\)ListNumberOpenAvg\.Aggreg\.Comp\.Cond\.NoisyPost\-proc\.SimpleAvg\.δ\\delta\-Mem0\.001\.943\.112\.2729\.4421\.1444\.4038\.4059\.0045\.5839\.84Temp\-LoRA\-4B0\.000\.005\.062\.5730\.7430\.4950\.8043\.6060\.6052\.0144\.92Temp\-LoRA\-9B0\.000\.005\.642\.8645\.2431\.9158\.4046\.4066\.8059\.0451\.42Temp\-LoRA\-27B0\.000\.005\.062\.5761\.9040\.2461\.0050\.6074\.2068\.6759\.45Metis\-4B1\.0814\.179\.9210\.2245\.0254\.6751\.0044\.6064\.8054\.6252\.54Metis\-9B0\.0024\.7215\.1816\.4930\.3034\.3553\.0043\.6066\.8054\.2247\.29Metis\-27B0\.0031\.3914\.5918\.5640\.6966\.0656\.6052\.8075\.4061\.4559\.04

As shown inTable[8](https://arxiv.org/html/2607.26760#S6.T8), Metis demonstrates strong OOD transfer on ATM\-Bench, consistently outperforming the memory baselines across model scales and most question types\. The advantage also holds for the deterministically scored number questions, indicating that the improvement is not merely an artifact of the LLM judge used for open\-ended questions\. Since ATM\-Bench requires models to retain and integrate heterogeneous evidence extracted from personal archives, these results suggest that the native memory procedure learned by Metis transfers beyond the patterns observed during training\. The results on MemDaily are more mixed: Metis remains competitive but does not consistently lead the memory baselines\. Together, the two benchmarks provide evidence that Metis’s native memory capability generalizes to benchmarks not used in constructing its training data\.

### 6\.5Memory Capacity Studies

In this part, we further explore the long\-term memory capability of Metis\. We evaluate this capability by modeling memory capacity, which comprises step\-level and trajectory\-level capacity\. Specifically, step\-level capacity refers to the maximum number of tokens accommodated within a single update\. Similarly, trajectory\-level capacity denotes the maximum number of update steps within an interaction trajectory\.

We construct a testing dataset, which contains 20 fictional users defined over a shared schema of 40 distinct and atomic persona domains\. First, the model generates 40 fine\-grained domains such as demographics, education and relationships\. Then, for each user, the domains are shuffled with a fixed seed and instantiated sequentially\. Previously generated attributes are provided as an immutable context to ensure logical consistency within each persona\. Each attribute is initially expanded into a biography\-style sentence conditioned on the complete persona\. After that, one direct question is generated for each domain and reused across all users\. Finally, each biography\-style message is rewritten as a concise first\-person statement that expresses only the corresponding fact\. During evaluation, the 40 records of each user form an ordered trajectory\. The statements are sequentially stored in memory, and the queries are used to probe the model\. In addition, the user attributes serve as the gold answer for LLM\-based judging\. Based on this dataset, we compare the performance of Qwen3\.5\-4B with full context and Metis\-4B under two evaluation settings\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x3.png)Figure 3:Results of memory capacity at the step\-level \(left\) and trajectory\-level \(right\)\.Step\-level Capacity\.This setting evaluates how much information the model can encode within a single memory update\. For each user of steptt, the memory state is reset, and the firstttstatements are concatenated and updated in one operation\. The model is then queried about the first, middle, and last facts in the updated content\. Since the full history is encoded from scratch at each step, this setting isolates the capacity of a single update as the input length increases\. The results are presented inFigure[3](https://arxiv.org/html/2607.26760#S6.F3)\(left\)\.

For the step\-level setting, Metis performs well when a single update contains only a small amount of information, but its accuracy decreases rapidly as the input becomes longer\. The performance on the first fact shows the clearest downward trend, while the middle and last facts exhibit larger fluctuations\. When the input exceeds several hundred words, performance at all three positions becomes low\. In contrast, the full\-context baseline remains much stronger, especially for the first fact\.

Trajectory\-level Capacity\.This setting evaluates how much information the model can retain over a sequence of memory updates\. For each user, the memory state is reset only once and then accumulates throughout the trajectory\. The 40 statements are divided into consecutive groups ofggstatements, withg=5g=5by default, and each group is concatenated and updated in Metis sequentially\. After every group\-level update, the model is queried about the first, middle, and current updated facts\. This setting measures the capacity of the evolving memory state under repeated updates, with performance reported against the number of updated trajectory steps\. The results are presented inFigure[3](https://arxiv.org/html/2607.26760#S6.F3)\(right\)\.

For the trajectory\-level setting, performance also declines as the number of updated steps increases\. The accuracy of the first fact decreases almost continuously, showing that early information is gradually weakened by later updates\. The middle and most recent facts also remain unstable, which suggests that new updates introduce interference throughout the whole memory state rather than only overwriting the oldest information\. Although the amount of information in each update is fixed, performance drops clearly as the trajectory becomes longer\. This confirms that repeated state transitions and accumulated compression errors form another major limitation of native memory\.

### 6\.6General Capability Studies

Previous experiments have demonstrated the effectiveness of Metis on memory\-related tasks\. However, integrating native memory into the forward computation potentially influences the backbone’s original behavior, which possibly decreases its general capabilities\. In this part, we further explore how Metis performs on the general tasks compared with its original backbone\. We compare Metis\-4B with Qwen3\.5\-4B and report the performance difference between them\. Specifically, we design two settings to evaluate the general capabilities of Metis in different stages\. The first is theInitial Stage, which measures performance on general tasks before any information is stored in memory, corresponding to stept=1t=1\. In this setting, Metis is reset to an empty memory state before receiving the original prompt in benchmarks, and the backbone receives the same prompt\. The second is theActive Stage, which evaluates Metis after it has accumulated irrelevant information over previous interaction steps, where we have the stept\>1t\>1\. For Metis, we reset the memory state and store task\-irrelevant messages before providing the benchmark prompt\. For the backbone, we prepend the same messages to its prompt\. Our experiments are conducted under MMLU\-Pro\[wang2024mmlu\], IFEval\[zhou2023instruction\], GSM8K\[cobbe2021training\], and MMMLU\[hendrycks2020measuring\]\. In IFEval, we adopt the strict evaluation setting, which verifies instruction compliance directly on the original model response without applying the response transformations used by the loose criterion\. The detailed prompts of irrelevant messages are provided inAppendix[F\.4](https://arxiv.org/html/2607.26760#A6.SS4)\. We present the results inTable[9](https://arxiv.org/html/2607.26760#S6.T9)\.

Table 9:Results of general capability tasks\. The gap is calculated as the performance of Metis\-4B minus that of Qwen3\.5\-4B\.BenchmarkInitial StageActive StageQwen3\.5\-4BMetis\-4BGapQwen3\.5\-4BMetis\-4BGapMMLU\-Pro46\.0045\.20−0\.80\-0\.8046\.0040\.90−5\.10\-5\.10IFEval79\.3079\.85\+0\.55\+0\.5576\.7154\.53−22\.18\-22\.18GSM8K83\.0982\.03−1\.06\-1\.0684\.5378\.92−5\.61\-5\.61MMMLU61\.3060\.50−0\.80\-0\.8059\.9056\.60−3\.30\-3\.30

The results show that Metis largely preserves the general capabilities of its original backbone at the initial stage\. It shows only minor decreases on the other tasks\. This indicates that the added memory architecture and memory\-specific training do not substantially change the model’s behavior when the memory state is empty\. A different trend appears at the active stage\. After irrelevant information is stored, Metis shows consistent performance drops across all benchmarks\. The degradation is moderate on MMLU\-Pro, GSM8K, and MMMLU, but is much larger on IFEval\. This suggests that irrelevant native memory may introduce noise into the forward computation and interfere with the processing of the current general task, especially on strict instruction following\. Overall, Metis retains most of its original general capability before memory is activated, but drops as more information is stored in the memory states\.

### 6\.7Low\-rank Decomposition

Storage overhead is a key efficiency metric for memory\. In short\-term tasks with limited information, parametric memory representations can be further compressed\. Therefore, to explore the storage optimization potential of Metis, we apply low\-rank decomposition to the memory states for efficient storage and reconstruct them before memory utilization\. Specifically, we cast the memory state𝐌t\(l\)\\mathbf\{M\}^\{\(l\)\}\_\{t\}to FP32 and compute an SVD along its last two dimensions\. Therefore, for the retained rankkk, the decomposition process can be represented as

𝐌t\(l\)=𝐔t\(l\)​𝚺t\(l\)​𝐕t\(l\)⊤\.\\mathbf\{M\}^\{\(l\)\}\_\{t\}=\\mathbf\{U\}\_\{t\}^\{\(l\)\}\\mathbf\{\\Sigma\}\_\{t\}^\{\(l\)\}\\mathbf\{V\}\_\{t\}^\{\(l\)\\top\}\.We maintain the low\-rank approximation𝐔^t\(l\)=𝐔t,:,1:k\(l\)\\hat\{\\mathbf\{U\}\}\_\{t\}^\{\(l\)\}=\\mathbf\{U\}\_\{t,:,1:k\}^\{\(l\)\},𝚺^t\(l\)=𝚺t,1:k,1:k\(l\)\\hat\{\\mathbf\{\\Sigma\}\}\_\{t\}^\{\(l\)\}=\\mathbf\{\\Sigma\}\_\{t,1:k,1:k\}^\{\(l\)\}, and𝐕^t\(l\)=𝐕t,:,1:k\(l\)\\hat\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\}=\\mathbf\{V\}\_\{t,:,1:k\}^\{\(l\)\}instead of the original full\-rank factors\. Here,𝐔t,:,1:k\(l\)\\mathbf\{U\}\_\{t,:,1:k\}^\{\(l\)\}and𝐕t,:,1:k\(l\)\\mathbf\{V\}\_\{t,:,1:k\}^\{\(l\)\}denote the firstkkcolumns of𝐔t\(l\)\\mathbf\{U\}\_\{t\}^\{\(l\)\}and𝐕t\(l\)\\mathbf\{V\}\_\{t\}^\{\(l\)\}, respectively, while𝚺t,1:k,1:k\(l\)\\mathbf\{\\Sigma\}\_\{t,1:k,1:k\}^\{\(l\)\}denotes the leadingk×kk\\times kdiagonal submatrix of𝚺t\(l\)\\mathbf\{\\Sigma\}\_\{t\}^\{\(l\)\}\. Then, the reconstruction can be expressed as

𝐌^t\(l\)=𝐔^t\(l\)​𝚺^t\(l\)​𝐕^t\(l\)⊤\.\\hat\{\\mathbf\{M\}\}^\{\(l\)\}\_\{t\}=\\hat\{\\mathbf\{U\}\}\_\{t\}^\{\(l\)\}\\hat\{\\mathbf\{\\Sigma\}\}\_\{t\}^\{\(l\)\}\\hat\{\\mathbf\{V\}\}\_\{t\}^\{\(l\)\\top\}\.We evaluatek∈\{1,4,16,64,128,256\}k\\in\\\{1,4,16,64,128,256\\\}on the four benchmarks with Metis\-4B aligned withSection[6\.2](https://arxiv.org/html/2607.26760#S6.SS2)\. We compare these models with low\-rank memory states to the original Metis models to characterize the trends of their performance degradation, where the full memory\-state dimension is 1024\. The results are presented inFigure[4](https://arxiv.org/html/2607.26760#S6.F4), which shows a clear trend as the retained rank increases\. Extremely low ranks, such as 1 and 4, cause substantial performance degradation\. This indicates that a few singular directions are insufficient to preserve the semantic information stored in the memory states\. Performance improves rapidly when the rank increases to 16 and becomes close to the full\-rank model at rank 64\. Further increasing the rank from 64 to 256 brings almost no additional improvement\. These results suggest that useful information in the memory states is mainly concentrated in a relatively low\-dimensional subspace\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x4.png)Figure 4:Results of low\-rank decomposition under different ranks of memory states\.Table 10:Results of low\-rank decomposition across different datasets\. Values in parentheses show recovery relative to Full, and the values above 100% are treated as judge variation rather than improvements\.Datasetk=1k=1k=4k=4k=16k=16k=64k=64k=128k=128k=256k=256FullLoCoMo \(Gold\)11\.31\(69\.4%\)10\.81\(66\.3%\)14\.21\(87\.1%\)16\.00\(98\.1%\)16\.36\(100\.3%\)16\.14\(99\.0%\)16\.31\(100\.0%\)NextMem22\.80\(54\.7%\)27\.83\(66\.7%\)38\.69\(92\.8%\)41\.43\(99\.4%\)41\.78\(100\.2%\)41\.71\(100\.0%\)41\.69\(100\.0%\)Metis Test17\.99\(31\.7%\)41\.26\(72\.7%\)53\.64\(94\.6%\)56\.63\(99\.8%\)56\.75\(100\.1%\)56\.04\(98\.8%\)56\.72\(100\.0%\)MemOps \(Gold\)5\.60\(31\.4%\)11\.49\(64\.4%\)18\.36\(102\.9%\)18\.36\(102\.9%\)18\.17\(101\.8%\)18\.79\(105\.3%\)17\.84\(100\.0%\)Overall14\.43\(43\.5%\)22\.84\(68\.9%\)31\.22\(94\.2%\)33\.10\(99\.9%\)33\.26\(100\.4%\)33\.17\(100\.1%\)33\.14\(100\.0%\)

Table[10](https://arxiv.org/html/2607.26760#S6.T10)further shows the sensitivity to low\-rank decomposition in different datasets\. The results show that different benchmarks have different levels of sensitivity to aggressive low\-rank compression\. At very small ranks, performance drops clearly on all datasets\. The decrease is particularly large on the Metis test set and MemOps \(Gold\), suggesting that memory operation tasks require sufficient representational capacity to preserve operation\-related information\. LoCoMo \(Gold\) is relatively less sensitive atk=1k=1, although its performance fluctuates at very small ranks\. This may indicate that some useful conversational information is concentrated in a small number of dominant directions\. However, these directions alone are not sufficient to support stable memory utilization\. MemOps \(Gold\) recovers its full\-rank performance atk=16k=16and remains stable at larger ranks\. In contrast, the Metis test set continues to improve fromk=16k=16tok=64k=64\. This difference suggests that the two memory operation benchmarks require different levels of memory capacity\. Across all datasets, performance becomes close to the full model atk=64k=64\. The overall recovery reaches 99\.9%, while further increasing the rank brings little additional benefit\. These results confirm that the memory states still contain substantial redundancy and that most useful information lies in a relatively low\-dimensional subspace\. They also show that the appropriate compression level depends on the task, since different benchmarks require different amounts and types of memory information\.

### 6\.8Case Studies

![Refer to caption](https://arxiv.org/html/2607.26760v1/x5.png)Figure 5:Case studies of Metis on different conversational scenarios\.We conduct qualitative case studies to show the behavior of Metis\-4B under different scenarios\. At each turn, the model first generates a response based on the current input and existing memory, and then updates the user input in the memory state\. We present several representative cases inFigure[5](https://arxiv.org/html/2607.26760#S6.F5)to qualitatively examine the native memory behaviors of Metis\-4B\. Each case starts from an empty native memory state, and the model must use information stored in previous interaction steps\. In the remembering case, Metis correctly stores Alice’s food preference and retrieves it in a later query\. In the multi\-fact case, the model retains several attributes about Alice and correctly selects her age after other attributes are introduced\. This result suggests that Metis can bind different values to their corresponding attributes and reduce interference among related facts\. The distractor case further shows that an unrelated dialogue turn does not overwrite the stored preference\. Metis can therefore distinguish useful memory from ordinary conversational content\.

The forgetting case demonstrates that the native memory state is not append\-only\. After receiving a forgetting instruction, Metis no longer provides the removed preference in the subsequent query\. This indicates that the model can modify its latent memory state according to the semantic intent of an instruction\. However, the immediate response to the forgetting instruction still repeats the old fact instead of explicitly confirming its removal\. The final memory state is correct, but the response at the operation step is not fully aligned with the intended memory operation\. This behavior may result from the current step over\-emphasizing previous memory states\. Overall, these cases show the effectiveness of Metis, while also revealing room for improvement in its consistency\.

## 7Related Work

### 7\.1Memory of LLMs and Agents

In recent years, large foundation models and agents have been widely applied to fields such as personal assistants\[lewis2020retrieval,zhang2026memsim\], deep research\[huang2025deep,zheng2025deepresearcher,du2025deepresearch\], and coding agents\[jiang2026survey,chen2021evaluating,roziere2023code\]\. A critical capability of these systems is memory, which stores past information to support future inference\[zhang2025survey\]\. Based on their representation forms, memory mechanisms of large foundation models and agents are generally categorized into three types, including textual memory, latent memory, and parametric memory\[zhang2025survey,hu2025memory\]\. Textual memory typically represents information as text, relying on RAG for storage and retrieval\. These methods provide information for backbones to support inference by In\-Context Learning \(ICL\)\[dong2024survey\]\. For example, MemoryBank\[zhong2024memorybank\]proposes a hierarchical storage approach with dual\-tower dense retrieval to maintain historical conversations with users\. MemTree\[rezazadeh2025isolated\]designs a tree\-structured memory mechanism to model the abstraction levels of information, which dynamically updates based on semantic embeddings\. In contrast, latent memory captures memory through intermediate activations of models\. For example, NextMem\[zhang2026nextmem\]compresses factual memory into latent representations through an autoregressive autoencoder, while MemGen\[zhang2025memgen\]generates latent memory tokens that are interwoven into the reasoning process\. Additionally, parametric memory injects knowledge into internal model parameters\. For example, Locas\[lu2026locas\]views the FFN as a soft look\-up table\. By adding a bypass FFN, it stores test\-time information from a key\-value perspective\. Furthermore, knowledge editing can also be considered a parametric memory method\[zhang2025survey\]\. ROME\[meng2022locating\]treats the projection matrix as an associative memory and inserts a new factual association through a rank\-one update\. Although textual memory remains the most effective approach in industry, latent memory and parametric memory are emerging as promising research directions\.

### 7\.2Fast Weight Programming

Recently, FWP has attracted widespread attention\. This paradigm not only uses parameters learned during training \(i\.e\.,slow weights\), but also maintains dynamic parameters \(i\.e\.,fast weights\) during inference to capture sequence\-dependent information\[ba2016using\]\. Existing methods in this line of work generally follow several main directions\. Linear attention replaces the softmax kernel with feature maps to achieve linear complexity and a recurrent state\[katharopoulos2020transformers\]\. In addition, it has been shown that linear transformers are secretly fast weight programmers\[schlag2021linear\]\. Subsequent works enrich the update rule, such as RetNet\[sun2023retentive\]and RWKV\[peng2023rwkv\]\. Furthermore, state space models compress a sequence into a fixed\-size recurrent state with linear\-time computation, such as S4\[gu2021efficiently\]and Mamba\[gu2023mamba\], while Mamba\-2\[dao2024transformers\]further reveals a duality between state space models and attention\. TTT also treats the recurrent state as fast weights that are optimized by self\-supervised gradient descent during inference, such as the TTT layer\[sun2024learning\]and Titans\[behrouz2026titans\]\.

### 7\.3Memory\-Augmented Neural Networks

MANNs introduce explicit memory modules to improve a model’s ability to store and retrieve task\-specific information during inference\. Early work, such as Memory Networks\[weston2014memory\]and Neural Turing Machines\[graves2014neural\], augments neural controllers with external memory and learns differentiable read and write operations over memory slots\. These methods show that neural models can use non\-parametric memory to support associative recall, algorithmic reasoning, and few\-shot adaptation\. However, their memory is usually maintained as a separate storage module, and the memory procedures are often designed independently from the backbone computation\. Recent models also maintain dynamic states during inference, such as recurrent memory\[bulatov2022recurrent\]\. Unlike static model parameters learned during training, these dynamic states are updated according to the current input sequence and capture information that changes over time\. Our work follows this general direction, but focuses on integrating memory storage and utilization directly into the model computation, so that the model can maintain sequence\-dependent information more natively\.

## 8Conclusion

In this paper, we introduce memory foundation models and provide formal definitions of native memory based on the memory state and memory procedures\. Based on this formulation, we propose Metis, the first prototype of memory foundation models\. We introduce Metis blocks composed of local memory blocks and hyper memory blocks, enabling the model to maintain compact dense memory states across interaction steps and to update them according to the current input and generated response\. We further construct a memory\-specific dataset from public benchmarks and design a mid\-training framework with memory reconstruction, memory operation, and regularization objectives\. Our experiments verify the effectiveness of Metis and analyze its behavior from multiple perspectives\.

Despite these promising results, Metis is still an early step toward memory foundation models\. Since the current native memory state compresses information into fixed\-size latent parameters, performance may degrade in extremely long\-term scenarios, and semantically similar facts may sometimes be confused in the latent space\. Therefore, native memory still cannot be viewed as a complete replacement for external memory\. Instead, we believe it opens a complementary direction for building future foundation models with more efficient, optimizable, and deeply integrated memory capabilities\. Future work may further improve memory capacity, controllability, and interpretability, explore hybrid systems that combine native and external memory, and scale native memory training to broader domains and longer interactions\.

## References

## Appendix ARoadmap for Memory Foundation Models

Metis represents an initial exploration of memory foundation models\. It transforms memory from an external information\-management module into a persistent internal state, integrating memory storage and utilization directly into the forward computation of the model\. The significance of native memory, however, extends beyond improving information retention capability\. In the longer term, it may reshape the computational paradigm, learning process, cognitive structure, and capability development of foundation models\. As illustrated inFigure[6](https://arxiv.org/html/2607.26760#A1.F6), we envision five progressive levels of capabilities in the development of memory foundation models:stateful capability,self\-managing capability,experience\-learning capability,persistent cognitive capability, andself\-evolving capability\. These capabilities characterize how memory may become progressively integrated into the foundation model itself, progressing from persistent state to autonomous memory organization, experience\-driven learning, persistent cognition, and continual capability evolution\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x6.png)Figure 6:Roadmap for memory foundation models\.Native memory aims to progressively transform foundation models from stateless predictors into stateful learners and, ultimately, into models capable of continual self\-evolution\. The five capabilities represent increasingly deep changes to the model’s computation, memory organization, learning process, cognitive representations, and capability formation\. Their development is supported by advances in memory architecture, learning objectives, scalable training, interpretability and control, and long\-horizon evaluation\.Level I: Stateful Capability\.Most existing foundation models remain fundamentally stateless conditional predictors\. At each inference step, their outputs are determined by fixed model parameters and the current context, while continuity across interactions is primarily maintained by repeatedly supplying historical information as external context\. Therefore, the first level is a transition from astatic function to a persistent state\. A memory foundation model maintains a dynamic internal state across inference steps\. Its output is jointly determined by the current input and its previous state, while each interaction updates the state used in subsequent computation:

\(Yt,𝐌t\+1\)=fΦ​\(Xt,𝐌t\),\(Y\_\{t\},\\mathbf\{M\}\_\{t\+1\}\)=f\_\{\\Phi\}\(X\_\{t\},\\mathbf\{M\}\_\{t\}\),whereΦ\\Phidenotes the static model parameters and𝐌t\\mathbf\{M\}\_\{t\}denotes the native memory state at steptt\. Unlike textual context supplied from outside the model,𝐌t\\mathbf\{M\}\_\{t\}is directly coupled with the model’s internal computation and evolves continuously during interaction\. This transition changes the basic computational unit of a foundation model\. The model is no longer merely a static mapping from inputs to outputs, but rather a stateful computational system that evolves over time\.

Level II: Self\-Managing Capability\.Possessing a persistent state does not by itself constitute a complete memory capability\. Information observed in real environments differs in importance, validity, abstraction level, time scale, and security requirements\. A model must therefore learn how to organize and maintain its internal state autonomously\. The second level moves frompersistent state to a learned memory lifecycle\. The model should determine what to remember, update, consolidate, and forget according to the semantics of incoming information, its expected future utility, and applicable privacy and safety constraints\. At this level, memory operations are no longer implemented primarily through external rules or discrete workflows\. Instead, both the memory state and the procedures that transform it become native, trainable components of the foundation model\. This enables memory to be selected, organized, and evolved within the model’s continuous computational space\.

Level III: Experience\-Learning Capability\.Once a model can autonomously maintain memory, the role of memory can expand from information support to capability adaptation\. The model should not only remember what happened, but also change as a consequence of what it has experienced\. The third level marks a transition frommemory utilization to experience\-driven learning\. Interaction histories become reward signals that the model can use to refine representations, knowledge, and behavioral regularities\. Experiences involving success, failure, feedback, or environmental change can be transformed into reusable internal capabilities rather than remaining isolated records\. This direction may gradually connect pre\-training, in\-context learning, test\-time adaptation, and continual learning within a unified framework\. Instead of remaining completely fixed after training, a foundation model could continuously adapt to new users, tasks, and environments while preserving previously acquired capabilities\.

Level IV: Persistent Cognitive Capability\.Experience learning explains how a model may adapt through interaction, but more advanced intelligence requires the formation of structured, persistent, and continually updated internal cognition\. The next level moves fromaccumulated experience to enduring internal models\. A foundation model should maintain evolving representations of the world, users, tasks, time, and itself\. These representations should capture temporal changes, causal dependencies, uncertainty, and conflicts between new evidence and existing beliefs\. When the environment changes, the model should be able to revise its internal representations while preserving global consistency\. At this level, memory no longer consists of disconnected pieces of historical information\. It becomes the substrate through which the model maintains cognitive continuity over extended periods\. Planning, personalization, and complex decision\-making may emerge as downstream expressions of this capability, but the defining transformation occurs within the foundation model’s internal cognitive representations rather than in an external agent workflow\.

Level V: Self\-Evolving Capability\.The long\-term objective of memory foundation models is to convert accumulated experience into the continual development of the model’s own capabilities\. A model should not only adapt its current state, but also reflect on, abstract, and reorganize past experience to discover new knowledge structures and learning strategies\. The final level represents a transition fromlocal adaptation to open\-ended capability evolution\. The model identifies experiences with long\-term value, abstracts transferable knowledge from specific interactions, and incorporates the resulting insights into future learning\. Then, the exploration and exploitation will constitute a continual feedback loop between memory foundation models and the environment\.

Outlook\.Memory foundation models should not be viewed as conventional foundation models equipped with a stronger storage module\. They point toward a more fundamental change in the paradigm of the foundation model\. Under this view, memory may become a foundational mechanism connecting computation, learning, cognition, and continual self\-evolution\.

A POSSIBLE PARADIGM SHIFTFoundation models may evolve fromstateless predictorsintostateful learners, and ultimately into learning systems that autonomously organize memory, maintain persistent cognition, and develop new capabilities through accumulated experience\.

## Appendix BExtensive Experiment Results

The results of memory operation tasks on MemOps \(Full\) are presented inTable[11](https://arxiv.org/html/2607.26760#A2.T11)\. According to the results, access to complete textual evidence remains the strongest setting\. Full\-context models achieve consistently high performance across all four operations, with moderate overall gains from increasing the backbone size\. In contrast, performance drops sharply when only partial context is available\. Standard Qwen models obtain near\-zero scores without context, confirming that these operations cannot be performed reliably using backbone knowledge alone\. The partial\-context results also show that increasing model size cannot compensate for missing historical evidence\.

Among no\-context methods, Metis\-27B achieves the best overall performance\. It obtains the highest scores on updating and reflection, while Metis\-9B performs best on forgetting\. In particular, the substantial improvement on reflection suggests that native memory can support higher\-level reasoning over stored information rather than only preserving individual facts\. Temp\-LoRA remains strongest on remembering, indicating that temporary parameter adaptation is effective for direct information retention\. However, Metis provides a more balanced advantage across memory operations and outperforms Temp\-LoRA in the overall average\.

Scaling Metis from 4B to 9B produces only moderate improvements, whereas Metis\-27B substantially increases the average score\. This demonstrates that a larger backbone can strengthen native memory operations when sufficient model capacity is available\. However, the gains are not uniform across operations\. For example, Metis\-27B improves updating and reflection but performs worse than Metis\-9B on forgetting\. This suggests that different memory operations may require different mechanisms and may not benefit equally from backbone scaling\. Despite these improvements, a large gap from the full\-context setting remains, showing that accurately storing, modifying, and reasoning over complex histories is still challenging\.

Table 11:The performance on memory operation tasks under MemOps \(Full\)\. Full\-context and partial\-context results are shown in gray to visually distinguish context\-access settings from the no\-context comparison\. Within the No Context setting, the best and second\-best scores areboldedandunderlined, respectively\. Average represents the micro\-average performance\.TypeMethodRememberUpdateForgetReflectAverageFull ContextQwen3\.5\-4B87\.0585\.4281\.3679\.6683\.52Qwen3\.5\-9B89\.5889\.5877\.9584\.1485\.69Qwen3\.5\-27B91\.5292\.1381\.8282\.5987\.19Partial ContextQwen3\.5\-4B31\.5523\.1520\.2314\.4822\.83Qwen3\.5\-9B25\.3012\.9615\.237\.2415\.77Qwen3\.5\-27B28\.4215\.2817\.5010\.1718\.50No ContextQwen3\.5\-4B3\.570\.001\.820\.001\.51Qwen3\.5\-9B3\.571\.850\.680\.001\.65Qwen3\.5\-27B4\.170\.930\.680\.691\.84Temp\-LoRA\-4B14\.8814\.352\.956\.389\.98Temp\-LoRA\-9B19\.7914\.584\.097\.7612\.19Temp\-LoRA\-27B20\.6810\.882\.505\.6910\.83δ\\delta\-Mem4\.465\.091\.361\.213\.06Metis\-4B10\.4213\.664\.779\.669\.70Metis\-9B13\.8414\.817\.279\.6611\.53Metis\-27B17\.2617\.133\.8620\.5215\.35

## Appendix CUpdate Designs across Model Scales

As discussed inSection[6\.3](https://arxiv.org/html/2607.26760#S6.SS3), using a linear update \(LU\) to replace the GDN\-based update \(GDU\) yields competitive performance at the 4B scale\. In this part, we further explore their performance at the 9B and 27B scales\. During the training phase, the Metis\-9B LU did not show sharp fluctuations on the validation curve, so it was trained to 14k steps without early stopping\. The reported Metis\-9B GDU result is based on the 8k checkpoint\.

Across all three scales, LU scores higher on the Metis test set but lower on LoCoMo \(Gold\)\. This recurring split points to a task\-dependent trade\-off between direct memory operations and long conversational memory\. The aggregate comparison is driven by the Metis test set at 9B and LoCoMo \(Gold\) at 27B, while the two updates remain nearly tied at 4B\. We also find that LU has a sharp drop in LoCoMo \(Gold\), which may indicate that LU is more vulnerable over long conversational trajectories\. In addition, LU consistently performs better on the Metis test set, possibly because its simpler update rule is easier to fit to the memory operations emphasized during training\. Both update rules achieve higher Overall at larger scales in the reported checkpoints, but the single\-run results, together with the unmatched 9B checkpoints and candidate 27B checkpoints, are insufficient to establish a general scaling law or statistical advantage\.

Table 12:Performance comparison of Metis with LU and GDU across different model scales\. Overall is the equal\-weight macro\-average of the four benchmarks\.ScaleUpdateLoCoMo \(Gold\)NextMemMetis Test SetMemOps \(Gold\)Overall4BGDU16\.3141\.6956\.7217\.8433\.14LU11\.9742\.7858\.5418\.5032\.959BGDU16\.8143\.3957\.9219\.6334\.44LU15\.3740\.1668\.6618\.1735\.5927BGDU26\.7450\.8273\.7724\.7644\.02LU14\.1652\.0975\.3224\.4441\.50

## Appendix DTransfer to a Llama Backbone

Architecturally, Metis can in principle be applied to various compatible causal decoder\-only Transformer backbones by integrating its native\-memory components into their Transformer layers\. Since the main experiments use Qwen3\.5 backbones, we test this architectural flexibility by applying Metis to Llama\-3\.1\-8B\.

Setup\.We use Llama\-3\.1\-8B as the backbone and follow the same training and evaluation setup as the main experiments, training the model for 14,000 steps\. The evaluation uses the same benchmark datasets as the main experiments\. We compare the resulting model with the same Llama backbone under no\-context and full\-context conditions, as well as the Qwen3\.5\-based Metis\-4B and Metis\-9B results from the main evaluation\.

Table 13:Backbone\-transfer evaluation\. Overall is their equal\-weight macro\-average\.MethodLoCoMo \(Gold\)NextMemMetis Test SetMemOps \(Gold\)OverallLlama\-3\.1\-8B \(No Context\)0\.2519\.2719\.983\.2510\.69Llama\-3\.1\-8B \(Full Context\)62\.8075\.7573\.4270\.4870\.61Metis \(Llama\-3\.1\-8B\)20\.5639\.9872\.3311\.2536\.03Metis\-4B16\.3141\.6956\.7217\.8433\.14Metis\-9B16\.8143\.3957\.9219\.6334\.44

Results\.As shown inTable[13](https://arxiv.org/html/2607.26760#A4.T13), Metis \(Llama\-3\.1\-8B\) exhibits the same overall pattern as in the main experiments: it outperforms the Llama no\-context control on all four benchmarks while still trailing the full\-context control\. It also achieves a slightly higher Overall score than both Qwen\-based Metis variants\. However, this advantage is not consistent across tasks: the Llama\-based variant outperforms both Qwen\-based variants on LoCoMo \(Gold\) and Metis test set but underperforms both on NextMem and MemOps \(Gold\)\. This task\-dependent divergence suggests that backbone choice shapes the task\-level performance profile even when the framework transfers successfully, supporting applicability across model families without implying backbone\-independent behavior or universal superiority\.

## Appendix EEvaluation Implementation Details

DenseRAG\.We construct the retrieval corpus from only the context visible to the current test instance\. The context is divided at sentence boundaries, and sentences longer than 256 embedding\-model tokens are further split into contiguous chunks\. Retrieval is performed by cosine similarity, and the top\-55chunks are provided to the corresponding Qwen3\.5\-4B, 9B, or 27B generator\. Gold answers, evidence identifiers, and future turns are never included in the retrieval corpus\.

Temp\-LoRA\.Following the official repository’s raw\-text adaptation design, we implement an official\-like memory\-task adaptation of Temp\-LoRA for Qwen3\.5\-4B, 9B, and 27B\. For each test instance, the LoRA and optimizer are reset, and each memory step is capped at 4,096 tokens \(enough for reported benchmarks\) and split into 1,024\-token chunks\. We perform two updates per chunk in BF16 with batch size 1 and AdamW \(learning rate5×10−55\\times 10^\{\-5\}, zero weight decay, and no scheduler\)\. The LoRA uses rank 64, scaling factor 64, and dropout 0\.05, and is applied to the attention and feed\-forward projections\. Its parameters and optimizer state persist within an instance and are discarded before the next instance\.

δ\\delta\-Mem\.We use the officially releaseddelta\-mem\_qwen3\_4b\-instructadapter with the Qwen3\-4B\-Instruct\-2507 backbone and the officialδ\\delta\-Mem runtime\. Each raw memory step is ingested as a separate user message\. Before issuing the question\-only query, we retain only the onlineδ\\delta\-Mem state and clear the chat history, processed input IDs, and KV cache\. We resetδ\\delta\-Mem before each instance\.

Metis\.We reset Metis to an emptyLocalMemorystate before each instance\. During the query phase, Metis receives only the question prompt\.

## Appendix FPrompts

### F\.1Prompt Notation and Coverage

Double braces, such as\{\{question\}\}, denote values inserted at runtime\. Only prompt text is shown verbatim\. Per\-instance questions, contexts, retrieved chunks, and answers are omitted\. Automatic wrapping inside the prompt boxes is typographical only\. Ablation and LowRank runs add no natural\-language prompt\.

### F\.2Information and Query Prompts of Baselines

#### F\.2\.1Qwen3\.5 Backbone

Query of Memory\-based QA Tasks \(No\-context\)\.

Question: \{\{question\}\}Answer with a short phrase\. If the answer is not known from the given information, say "No information available"\.

Query of Memory Operation Tasks \(No\-context\)\.

Question: \{\{question\}\}Answer the question using the memory context\. Be concise, but include all necessary details\. If the answer is not known from the given information, say "No information available"\.

Query of Memory\-based QA Tasks \(Partial\-context\)\.

Optional metadata fields are emitted only when available\.

Retrieved context:\[chunk 1 \| date=\{\{optional\_date\_time\}\} \| session=\{\{optional\_session\_id\}\} \| speaker=\{\{optional\_speaker\}\}\]\{\{retrieved\_chunk\_text\}\}Question: \{\{question\}\}Answer with a short phrase using only the retrieved context\. If the answer is not known from the retrieved context, say "No information available"\.Short answer:

Query of Memory Operation Tasks \(Partial\-context\)\.

Retrieved context:\[chunk 1\]\{\{retrieved\_chunk\_text\}\}Question: \{\{question\}\}Answer with a short phrase using only the retrieved context\. If the answer is not known from the retrieved context, say "No information available"\.Short answer:

Query of LoCoMo \(Gold\) in Memory\-based QA Tasks \(Full\-context\)\.

Evidence\-session context:SESSION: \{\{session\_id\}\}DATE: \{\{date\_time\}\}\{\{dialogue\_turn\_id\}\} \{\{speaker\}\} said: "\{\{dialogue\_text\}\}" Shared image caption: \{\{optional\_image\_caption\}\}Question: \{\{question\}\}Answer with a short phrase\. If the answer is not known from the given information, say "No information available"\.

Query of NextMem in Memory\-based QA Tasks \(Full\-context\)\.

Reference context:\{\{reference\_context\}\}Question: \{\{question\}\}Answer with a short phrase\. If the answer is not known from the given information, say "No information available"\.

Query of Memory Operation Tasks \(Full\-context\)\.

The context block repeats for each normalized context item\.

Memory context:\{\{context\_id\}\}:\{\{memory\_context\}\}Question: \{\{question\}\}Answer the question using the memory context\. Be concise, but include all necessary details\. If the answer is not known from the given information, say "No information available"\.

#### F\.2\.2Metis, Temp\-LoRA, andδ\\delta\-Mem

During the information stage, we retain each method’s method\-specific write interface while keeping the ordered memory\-step contents fixed\. Metis prepends a fixed, answer\-independent commit instruction to each memory step before its memory\-commit operation\. Temp\-LoRA updates its temporary LoRA directly on the memory context\.δ\\delta\-Mem passes each raw memory step as a user message through the officially released runtime’s native chat\-message ingestion path to update its online state\. At query time, all three methods use the same prompt\.

Share Query of Metis, Temp\-LoRA, andδ\\delta\-Mem\.

Answer from the learned memory state produced during the information phase\.Give the shortest factual answer you can\. Do not explain\.Question: \{\{question\}\}Short answer:

Information\-stage Prompt of Metis in Memory\-based QA Tasks\.

Conversation memory segment\.Commit the following dated dialogue segment to memory for later question answering\.\{\{memory\_step\.content\}\}

Information\-stage Prompt of Metis in Memory Operation Tasks\.

Conversation memory segment\.Commit the following dialogue segment to memory for later question answering\.\{\{memory\_step\.content\}\}

Information\-stage Payload of Temp\-LoRA andδ\\delta\-Mem\.

These methods passmemory\_steps\[\*\]\.contentdirectly during the information step\.

\{\{memory\_step\.content\}\}

### F\.3Prompts for LLM\-as\-a\-Judge

Formal scoring usesgpt\-4\.1\-miniwith temperature 0, three repeats\. For each example, the score is the median of the three LLM\-as\-a\-judge scores\.

System Message\.

You are a conservative but fair evaluator for a memory question\-answering benchmark\. Your job is to avoid overly generous partial credit while still accepting truly equivalent answers, aliases, abbreviations, and harmless formatting differences\. Return JSON only\.

User Instruction\.

Grade model\_output against gold\_answer for the question\. Return JSON with keys: score \(0 to 1\), pass \(boolean\), matched\_points \(array of strings\), missed\_points \(array of strings\), and rationale \(short string\)\. Use this strict rubric: give 1\.0 only when the answer contains the correct core entity/value/date/relationship asked for, allowing aliases and semantically equivalent wording\. Give 0\.5 to 0\.75 only when the output includes the correct core answer but has minor extra wording, minor imprecision, or one secondary omission\. Give 0 for a different person, organization, place, number, date, title, relation, or answer choice; for a broad category when the gold answer is a specific entity; for answers that merely share common words with gold; for plausible guesses unsupported by the exact answer; or when the model says the answer is unknown/unavailable while gold is answerable\. Do not reward explanation quality if the final answer is wrong\. If the question asks for a country/state/type and the model gives exactly that correct country/state/type, it is correct even if it could be guessed from world knowledge\.

User Payload \(JSON Schema\)\.

\{"instruction": "\{\{judge\_instruction\_above\}\}","question": "\{\{question\}\}","gold\_answer": "\{\{gold\_answer\}\}","model\_output": "\{\{model\_output\}\}","raw\_category": "\{\{raw\_category\}\}","is\_adversarial": "\{\{is\_adversarial\}\}","baseline": "\{\{baseline\}\}"\}

### F\.4Prompts in General Capability Study

The active stage setting uses the same message for both models\. For Qwen3\.5\-4B, it is prepended to the complete benchmark prompt\. For Metis\-4B, the model is reset for each example, the message is stored in the memory state, and the unchanged benchmark prompt is then provided\.

This session is for general\-purpose evaluation\. For any upcoming question, follow the instruction closely, reason carefully when needed, and answer based on the information available\.

MMLU\-Pro, IFEval, GSM8K, and MMMLU retain their native per\-instance benchmark prompts\.

## Appendix GEfficiency

This appendix evaluates the inference efficiency of Metis at the 4B scale, characterizing both the benefits of native memory and the costs it introduces\. We first measure application\-level end\-to\-end and query latency on LoCoMo \(Gold\)\. Furthermore, we isolate the effect of history through a controlled sweep over 512 to 128K context tokens, with latency decomposed into fine\-grained stage\-level components\. Finally, beyond latency, we examine the storage efficiency of maintaining a fixed\-size memory state, measuring the storage that each method must persist per session as history grows\.

### G\.1End\-to\-End Latency on LoCoMo \(Gold\)

Evaluation Protocol\.We evaluate all methods on the 1,527 examples in the LoCoMo \(Gold\) evidence\-session setting, following the baseline configurations described inSection[6](https://arxiv.org/html/2607.26760#S6)\. Each example is evaluated independently under a cold\-start protocol: the method\-specific state is reset, the evidence session is processed, and the question is then answered without reusing state across examples\. All methods use greedy decoding with at most 96 generated tokens\. Each run includes one unmeasured warm\-up example, and CUDA synchronization is applied at every timing boundary\. End\-to\-end latency includes retrieval, test\-time adaptation, and memory writing when required, followed by query generation\. Query latency starts after these preparation operations have completed\. Model loading and answer evaluation are excluded\. Because generation uses natural stopping behavior, the results measure application\-level latency rather than controlled decode\-only latency\.

Table 14:End\-to\-end latency, query latency, input length, generation length, and effective throughput on the LoCoMo \(Gold\) evidence\-session setting\. The smallest and second\-smallest latency values areboldedandunderlined, respectively\.MethodE2ELatency \(s\)QueryLatency \(s\)Prompt Tokens\(Avg\./P95\)Committed Tokens\(Avg\./P95\)GeneratedTokensEffectiveThroughput\(tokens/s\)Avg\.P95Avg\.P95Avg\.P95No Context0\.1490\.1450\.1490\.14550\.2 / 58\.0–3\.03\.020\.16Full Context0\.6073\.0120\.6073\.0121410\.6 / 3345\.8–5\.716\.09\.37Partial Context0\.2680\.4560\.2230\.412290\.5 / 322\.0–4\.011\.017\.73Temp\-LoRA1\.5673\.2920\.3050\.74656\.2 / 64\.01386\.0 / 3365\.25\.215\.017\.11δ\\delta\-Mem0\.8841\.6000\.6561\.36549\.2 / 57\.01377\.2 / 3353\.18\.821\.013\.44Metis0\.5620\.9260\.3600\.60956\.2 / 64\.01449\.0 / 3515\.48\.315\.023\.15

Results\.Partial Context has the lowest end\-to\-end latency among methods that use historical information, with an average of 0\.268 seconds and a P95 of 0\.456 seconds\. Metis achieves an average latency of 0\.562 seconds, slightly below the 0\.607 seconds of Full Context, while reducing P95 latency from 3\.012 to 0\.926 seconds, a reduction of 69\.3%\. Compared withδ\\delta\-Mem, Metis reduces average and P95 end\-to\-end latency by 36\.4% and 42\.2%, respectively; compared with Temp\-LoRA, the corresponding reductions are 64\.1% and 71\.9%\. Metis also reduces the average query latency from 0\.656 to 0\.360 seconds relative toδ\\delta\-Mem, with P95 decreasing from 1\.365 to 0\.609 seconds\.

Analysis\.The slightly lower average latency and substantially lower P95 latency relative to Full Context indicate that Metis provides a more stable query path across LoCoMo \(Gold\) instances with varying history lengths\. Full Context processes 1,410\.6 prompt tokens on average and 3,345\.8 tokens at P95 for every query\. Metis commits a similar amount of historical information, at 1,449\.0 tokens on average and 3,515\.4 tokens at P95, but its subsequent query contains only 56\.2 prompt tokens on average and 64 tokens at P95\. Thus, Metis shifts historical\-information processing into a separate memory\-write stage and avoids replaying the original evidence during querying, contributing to its lower query and tail latency\. Its advantage overδ\\delta\-Mem comes mainly from memory utilization rather than memory construction: its average write latency is moderately lower at 0\.199 versus 0\.227 seconds, whereas it reduces average and P95 query latency by 45\.2% and 55\.4%, respectively\. Temp\-LoRA, in contrast, is dominated by its 1\.186\-second test\-time adaptation\. These comparisons show that Metis has lower online overhead than the two state\-based parametric memory baselines\. Partial Context remains the fastest cold\-start method because its retrieval stage costs only 0\.044 seconds on average and produces a compact 290\.5\-token query prompt\. The token statistics ofδ\\delta\-Mem follow its Qwen3 tokenizer, while the other methods use the Qwen3\.5 tokenizer\.

### G\.2Latency Scaling with Context Length

Evaluation Protocol\.We evaluate all six methods at the 4B scale on a single NVIDIA A800 GPU using controlled histories from 512 to 128K tokens\. Each final query contains 512 content tokens, excluding method\-specific templates, and generates either 32 or 128 tokens with batch size 1\. For every method, context length, and output length, we report the mean and standard deviation over five measured runs after one warm\-up run to remove kernel initialization\. Metis processes the history in chunks of at most 2K tokens, so its number of commits grows naturally with context length\. Partial Context rebuilds its index over the complete history for every run and retrieves the top five sentence chunks\. This is a generation\-only experiment without answer scoring\. End\-to\-end latency includes all method\-specific history preparation or writing and the subsequent query, whereas query latency is measured after history preparation has completed\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x7.png)Figure 7:End\-to\-end latency as controlled context length increases, with 32 generated tokens \(left\) and 128 generated tokens \(right\)\. Metis E2E includes streaming history ingestion and query generation; Metis query excludes ingestion\. Readout latency is computed as Metis query latency minus No Context latency\. The first measured crossover between Metis E2E and Full Context occurs at 64K for both output lengths\.Results\.At context lengths up to 32K, Metis has slightly higher end\-to\-end latency than Full Context\. The ordering reverses at the first measured 64K point for both output lengths: for 32 generated tokens, Full Context and Metis take 5\.909 and 5\.390 seconds, respectively, giving a1\.096×1\.096\\timesspeedup; for 128 generated tokens, they take 10\.228 and 9\.522 seconds, giving a1\.074×1\.074\\timesspeedup\. At 128K, these E2E speedups increase to1\.497×1\.497\\times\(14\.254 versus 9\.521 seconds\) and1\.595×1\.595\\times\(21\.404 versus 13\.423 seconds\), respectively, inFigure[7](https://arxiv.org/html/2607.26760#A7.F7)\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x8.png)Figure 8:Normalized end\-to\-end latency breakdown across context lengths for 32\-token \(top\) and 128\-token \(bottom\) outputs\. For Metis, model forward/query prefill includes the history\-encoding forwards and final query prefill, while write/commit/adaptation contains only the pure memory update\. Forδ\\delta\-Mem and Temp\-LoRA, the latter category represents state writing and temporary adaptation, respectively; indexing and retrieval is reported separately for Partial Context\. Other includes reset, tokenization, prompt construction, synchronization, text decoding, and unseparated host\-side work\.Figure[8](https://arxiv.org/html/2607.26760#A7.F8)further shows that pure memory commit is a small fraction of Metis latency\. At 64K and 128K, commit takes approximately 0\.203 and 0\.406 seconds, respectively, and accounts for only 2\.1–4\.3% of Metis E2E latency across the two output lengths; most ingestion time instead comes from the history\-encoding backbone forwards, which grow from approximately 3\.86 to 7\.65 seconds\. Metis also has lower E2E latency than both state\-based baselines at every measured context and output length\. At 128K, it is2\.402\.40–2\.67×2\.67\\timesfaster thanδ\\delta\-Mem and6\.466\.46–8\.71×8\.71\\timesfaster than Temp\-LoRA\. Consistent with this gap, state writing contributes 71\.7–90\.8% ofδ\\delta\-Mem E2E latency at 128K, while temporary adaptation contributes 92\.2–97\.7% for Temp\-LoRA\.

Analysis\.The E2E results reveal two complementary efficiency advantages\. Relative to Full Context, Metis becomes increasingly advantageous as history grows in the long\-context regime\. The crossover is first observed only at 64K because Qwen3\.5\-4B already uses linear attention in 24 of its 32 layers, substantially reducing the context\-dependent KV traffic of Full Context and thereby delaying the E2E crossover in favor of Metis\. As the KV caches of the remaining 8 full\-attention layers continue to grow, the advantage of Metis becomes more pronounced: when context doubles from 64K to 128K, Full Context prefill increases by approximately2\.67×2\.67\\times\(4\.34 to 11\.60 seconds\) and decode by approximately1\.65×1\.65\\times\(1\.40 to 2\.31 seconds for 32\-token outputs and 5\.74 to 9\.45 seconds for 128\-token outputs\), widening the E2E latency gap\. Relative to the two state\-based baselines, Metis maintains lower E2E latency throughout the entire context\-length sweep because it avoids their increasingly expensive state writing or temporary adaptation\. We use query latency only to isolate the cost of memory readout, estimated as Metis query latency minus No Context latency\. For 32\-token outputs, the estimated readout is 0\.091 seconds at 64K and 0\.106 seconds at 128K; for 128\-token outputs, it is 0\.770 and 0\.559 seconds, respectively\. These values account for only 7\.2–14\.4% of Metis query latency and do not increase when the history doubles\.Figure[8](https://arxiv.org/html/2607.26760#A7.F8)similarly shows that pure memory commit contributes only 2\.1–4\.3% of Metis E2E latency at 64K and 128K, with most ingestion time spent in the history\-encoding backbone forwards\. Thus, neither memory readout nor pure memory commit is a major latency bottleneck\. In principle, both stages can be executed concurrently with backbone attention on separate CUDA streams\. Such asynchronous overlap, however, requires finer\-grained kernel launches and inter\-stream synchronization, introducing launch and scheduling overhead that can offset the latency hidden at this scale\. We therefore do not enable this optimization in the final implementation\.

### G\.3Per\-Session Storage across Context Lengths

Evaluation Protocol\.We also measure the persistent per\-session storage of the 4B methods over different context lengths from 512 to 32K token\. All values are obtained from runtime objects, rather than estimated analytically from parameter counts\. For Full Context, we retain the KV cache required to resume a session without replaying its history; the measurement includes the controlled context, a fixed 512\-token query prompt, and 32 generated tokens\. KV cache and model states are stored in BF16\. The RAG store contains FP32 BGE\-M3 embeddings and the corresponding chunk text\. For state\-based methods, we count only state that must be persisted separately for each session\.

![Refer to caption](https://arxiv.org/html/2607.26760v1/x9.png)Figure 9:Persistent storage per session as a function of context length\.Results\.Full Context storage grows from 87\.06 MB at 512 context tokens to 1,118\.86 MB at 32K\. Its 8 full\-attention layers add 32\.8 KB of KV cache per token, while the 24 linear\-attention layers contribute a constant 51\.90 MB state\. In contrast, Metis Full remains at 16\.79 MB and Metisk=64k=64remains at 2\.11 MB\. At 32K, Full Context therefore occupies67×67\\timesthe Metis Full state and529×529\\timesthe Metisk=64k=64state; extrapolating the measured linear trend to 128K gives approximately256×256\\timesand2,000×2\{,\}000\\times, respectively\. The other constant\-size states are 4\.6 KB forδ\\delta\-Mem and 190\.3 MB for Temp\-LoRA, whereas RAG Store grows from 0\.104 MB at 512 tokens to 6\.051 MB at 32K\.

Analysis\.Metis eliminates context\-dependent storage growth: the Full Context KV cache reaches 1,066\.96 MB at 32K, whereas both Metis states remain constant\. Low\-rank persistence provides a particularly efficient operating point: Metis\(k=64\)\(k=64\)uses one eighth of the Metis Full state while retaining 99\.9% of full\-state performance on average\.δ\\delta\-Mem occupies only 4\.6 KB because its official rank\-8 configuration stores one8×88\\times 8BF16 online matrix in each of 36 attention modules, compressing each session into a low\-dimensional state; this limited capacity is consistent with its lower LoCoMo score \(Table[6](https://arxiv.org/html/2607.26760#S6.T6)\)\. Despite the small state, every query token reads it in all 36 attention modules and applies additionalq/k/v/oq/k/v/odelta projections, introducing per\-layer matrix operations and kernel launches on top of the backbone attention\. RAG Store also grows with history and depends on a separate retrieval and indexing pipeline; Metisk=64k=64becomes smaller at approximately 11K tokens and remains smaller at the measured 16K and 32K points\. Temp\-LoRA requires a 190\.3 MB per\-session adapter,11\.3×11\.3\\timesthe Metis Full state\. Overall, Metis provides high storage efficiency for long texts: both Metis Full andk=64k=64remain constant and substantially smaller than Full Context, whilek=64k=64also becomes smaller than RAG Store as history grows\. As a storage\-only upper bound on an 80 GB GPU with approximately 72 GB remaining after 4B model weights, 32K Full Context caches accommodate roughly 64 resident sessions, compared with about 4,000 full\-state Metis sessions; the former decreases with history length, while the latter is independent of it\.