NGM: A Plug-and-Play Training-Free Memory Module for LLMs

# NGM: A Plug-and-Play Training-Free Memory Module for LLMs
Source: [https://arxiv.org/html/2605.16893](https://arxiv.org/html/2605.16893)
Yuwen Qu\* Nanjing Universityyuwenqu@smail\.nju\.edu\.cnWenhui Dong\* Nanjing Universitywenhui\.dong@smail\.nju\.edu\.cnChenyang Si Nanjing Universitychenyang\.si@nju\.edu\.cnCaifeng Shan†\\dagger Nanjing Universitycfshan@nju\.edu\.cn

###### Abstract

Recent studies introduce conditional memory modules that decouple knowledge storage from neural computation, enabling more direct knowledge access\. Compared to MoE, which relies on dynamic computation paths, explicit lookup provides a more efficient knowledge retrieval mechanism\. However, these approaches still depend on learned memory embeddings, requiring additional training and limiting flexibility\. To address this, we proposeN\-gram Memory \(NGM\), a training\-free, plug\-and\-play module composed of aCausal N\-Gram Encoderand aCosine\-Gated Memory Injector\. The Causal N\-Gram Encoder directly averages the pretrained token embeddings of the backbone model to construct N\-gram representations, thereby eliminating the need to train separate N\-gram embeddings from scratch\. This design requires neither an additional memory table nor a retrieval pipeline\. The Cosine\-Gated Memory Injector then uses a non\-parametric cosine gate with ReLU to modulate the retrieved embeddings into the contextual representations\. We evaluate NGM on the Qwen3 series from 0\.6B to 14B across eight benchmarks\. NGM improves average performance by 0\.5 to 1\.2 points, with particularly clear gains on code generation and knowledge\-intensive tasks \(e\.g\., \+3\.0 on LiveCodeBench and \+3\.03 on GPQA for Qwen3\-14B\)\. Moreover, NGM also improves performance in multimodal benchmarks \(e\.g\., MMStar \+1\.53 on Qwen3\-VL\-2B\)\. Code is available at[https://github\.com/PioneerQyw/NGM](https://github.com/PioneerQyw/NGM)\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding authors\.## 1Introduction

Transformer\-based large language models \(LLMs\)\[[39](https://arxiv.org/html/2605.16893#bib.bib2)\]provide strong contextual modeling and semantic reasoning, yet language modeling combines two qualitatively different demands: dynamic compositional computation and the reuse of local, static, and stereotyped patterns\[[15](https://arxiv.org/html/2605.16893#bib.bib36),[10](https://arxiv.org/html/2605.16893#bib.bib37)\]\. Named entities, repeated identifiers, units, terminology, and formulaic phrases often behave less like problems requiring deep reasoning and more like patterns that could be recovered through inexpensive lookup\[[4](https://arxiv.org/html/2605.16893#bib.bib38),[29](https://arxiv.org/html/2605.16893#bib.bib39),[33](https://arxiv.org/html/2605.16893#bib.bib40)\]\. However, standard Transformers lack a native knowledge lookup primitive for such local lexical and symbolic dependencies, forcing LLMs to reconstruct them through attention and feed\-forward computation at inference time\[[40](https://arxiv.org/html/2605.16893#bib.bib35),[8](https://arxiv.org/html/2605.16893#bib.bib1)\]\.

Lookup\-style memory provides a natural way to separate static pattern reuse from dynamic Transformer computation\[[24](https://arxiv.org/html/2605.16893#bib.bib15),[40](https://arxiv.org/html/2605.16893#bib.bib35)\]\. Recent work has explored this direction by introducing explicit learned memory components: Engram\[[8](https://arxiv.org/html/2605.16893#bib.bib1)\]formulates*conditional memory*with learnedNN\-gram lookup tables and context\-dependent gating, while embedding\-scaling methods expand capacity through additional token\-level orNN\-gram embedding parameters\[[42](https://arxiv.org/html/2605.16893#bib.bib41),[38](https://arxiv.org/html/2605.16893#bib.bib43),[28](https://arxiv.org/html/2605.16893#bib.bib42),[12](https://arxiv.org/html/2605.16893#bib.bib44)\]\. These approaches demonstrate that local lookup is a useful axis for improving language models, but they obtain this benefit through additional trainable parameters, dedicated training, and in some cases specialized storage or retrieval infrastructure\.

This motivates our central research question:

> *Can already\-trained LLMs recover useful local\-memory benefits without retraining or adding learned memory tables?*

A typical lookup\-style memory pipeline first constructs trainedNN\-gram embeddings, retrieves a sparse subset of relevant memory entries, and then fuses the retrieved memory with hidden states through context\-aware gating\. The first obstacle in this pipeline is the need to train a separateNN\-gram embedding space\. We instead ask whether the backbone’s already\-trained token embeddings can be reused directly: by averaging pretrained token embeddings within a local causal window, we obtainNN\-gram features without introducing any new memory table\.

This simple construction is useful only if the aggregated N\-gram features remain compatible with the model’s hidden states\. As shown in Figure[1](https://arxiv.org/html/2605.16893#S1.F1), N\-gram embeddings align more strongly with Qwen3\-8B hidden states than both position\-shuffled N\-gram controls and random\-token controls across depth\. At the two default injection layers, the actual mean cosine similarities are 0\.312 and 0\.137, compared with 0\.172 and 0\.084 for shuffled controls and 0\.014 and 0\.008 for random controls\. This suggests that non\-parametrically aggregated N\-gram embeddings can be directly fused with hidden states through a training\-free cosine gate\.

![Refer to caption](https://arxiv.org/html/2605.16893v1/x1.png)Figure 1:Alignment between hidden states and aggeratingNN\-gram embedding in the Qwen3\-8B model\.Motivated by this view, we proposeNGM \(N\-gram Memory\), a training\-free, plug\-and\-play module that injects localNN\-gram signals into frozen decoder\-only LLMs\. The key idea is to treat the pretrained embedding space not only as an input interface, but also as a lightweight source of reusable local memory: if nearby tokens form stable lexical, symbolic, or phrase\-level patterns, their aggregated embeddings may provide a useful cue that the decoder can reuse instead of reconstructing entirely through deeper Transformer computation\. As shown in Figure[2](https://arxiv.org/html/2605.16893#S3.F2), NGM realizes this idea through two non\-parametric components: aCausal N\-gram Encoderand aCosine\-Gated Memory Injector\. Given an input sequence, the Causal N\-gram Encoder constructs causal multi\-scaleNN\-gram representations by aggregating the backbone’s pretrained token embeddings within local trailing windows, thereby capturing local patterns at different granularities without learning separate memory entries\. The Cosine\-Gated Memory Injector then compares these input\-derivedNN\-gram representations with decoder hidden states using a ReLU\-filtered cosine gate and writes the resulting memory update through a scaled residual connection, so that only positively aligned local\-memory signals are injected\. This design is meaningful from both practical and analytical perspectives\. In practice, it can be attached to already\-trained LLMs without additional parameters, external knowledge sources, or retrieval infrastructure\. From an analytical perspective, it provides a controlled way to test whether pretrained embedding spaces already contain exploitable local\-memory structure that can improve generation\.

We evaluate NGM on Qwen3 models ranging from 0\.6B to 14B across eight benchmarks covering mathematics, code, knowledge, and alignment\. Across all tested scales, NGM consistently improves the average score by \+0\.5 to \+1\.2 points, with the most pronounced gains observed on code generation and several knowledge\-intensive benchmarks, such as \+3\.0 on LiveCodeBench and \+3\.03 on GPQA for Qwen3\-14B\. In addition, we extend our method to multimodal tasks\. Results on Qwen3\-VL\-2B show that applying NGM only to the language decoder improves all reported benchmarks, demonstrating a certain degree of generality of our approach\.

## 2Related work

#### Conditional memory and embedding scaling\.

ClassicalNN\-gram models capture short\-range statistics through fixed\-order Markov assumptions\[[25](https://arxiv.org/html/2605.16893#bib.bib31),[7](https://arxiv.org/html/2605.16893#bib.bib32)\], and the insight that local lexical patterns carry strong predictive structure remains relevant in the neural era\[[32](https://arxiv.org/html/2605.16893#bib.bib33),[2](https://arxiv.org/html/2605.16893#bib.bib34)\]\. Mixture\-of\-Experts \(MoE\) models scale capacity through conditional computation\[[36](https://arxiv.org/html/2605.16893#bib.bib3),[16](https://arxiv.org/html/2605.16893#bib.bib4)\]; conditional memory explores a complementary sparsity axis based on lookup\. Recently, a wave of work has revived this intuition as*embedding scaling*, treatingNN\-gram or token\-level embedding tables as a dedicated parameter axis for expanding model capacity\. SCONE\[[42](https://arxiv.org/html/2605.16893#bib.bib41)\]trains an auxiliary transformer to produce contextualizedNN\-gram embeddings but relies on an auxiliary encoding model that introduces additional training FLOPs; L3\[[38](https://arxiv.org/html/2605.16893#bib.bib43)\]generalizes tokenizer embedding tables to decoder layers via static routing, yet requires learned per\-layer aggregation matrices and CPU\-offloaded storage; LongCat\-Flash\-Lite\[[28](https://arxiv.org/html/2605.16893#bib.bib42)\]scales hash\-basedNN\-gram embeddings beyond 30B parameters, demanding large\-scale distributed training and hash\-table infrastructure; and MeKi\[[12](https://arxiv.org/html/2605.16893#bib.bib44)\]injects token\-level memory experts re\-parameterized into static lookup tables, which still requires a dedicated training phase to learn the memory bank\. Most closely related is Engram\[[8](https://arxiv.org/html/2605.16893#bib.bib1)\], which formalizes*conditional memory*via hashedNN\-gram lookup with context\-aware gating and a sparsity allocation framework, scaling to 27B parameters with algorithm\-system co\-design for deep\-layer injection\. A related line augments language models with non\-parametric datastores or retrieval over hidden states and external corpora\[[24](https://arxiv.org/html/2605.16893#bib.bib15),[40](https://arxiv.org/html/2605.16893#bib.bib35),[19](https://arxiv.org/html/2605.16893#bib.bib17),[3](https://arxiv.org/html/2605.16893#bib.bib16)\]; by contrast, NGM reuses the backbone embedding matrix directly and does not build a datastore or retrieval index\. All of these approaches share a common requirement: training dedicated embedding parameters and, in most cases, specialized infrastructure for storage and retrieval\. NGM revisits the same intuition under a stricter constraint—it constructs causal multi\-scaleNN\-gram representations directly from the backbone’s existing token embeddings at inference time, requiring no additional training, no external memory tables, and no specialized infrastructure\.

#### Residual stream alignment\.

Work on Transformer interpretability has characterized the residual stream as a shared linear workspace for successive computation\[[14](https://arxiv.org/html/2605.16893#bib.bib45)\]\. The logit lens\[[34](https://arxiv.org/html/2605.16893#bib.bib46)\]and follow\-up probes\[[11](https://arxiv.org/html/2605.16893#bib.bib48)\]show that intermediate hidden states remain partially projectable into vocabulary space via the*unembedding*matrix \(the language\-modeling head\)\. For models with tied embeddings this directly implies alignment with the input embedding layer; for models with untied embeddings \(including the Qwen3 family used here\), the implication is indirect\. Our*residual alignment*argument therefore adopts a weaker, empirically grounded premise: hidden states retain enough geometric compatibility with the*input*embedding space for cosine similarity to serve as a useful training\-free gating signal\. We validate this premise in §[4\.5](https://arxiv.org/html/2605.16893#S4.SS5), where cosine similarity between hidden states and input\-derived N\-gram embeddings significantly exceeds both shuffled and random controls\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2605.16893v1/x2.png)Figure 2:Overview of NGM\. The Causal N\-gram Encoder constructs multi\-scaleNN\-gram representations from the backbone’s token embeddings; the Cosine\-Gated Memory Injector scores them against decoder hidden states and injects the aggregated residual into selected layers\.As illustrated in Figure[2](https://arxiv.org/html/2605.16893#S3.F2), NGM is a training\-free memory module that derives local memory signals directly from the backbone model’s token embedding matrix and injects them into frozen decoder representations through a non\-parametric cosine gate\. The module contains two components\. The*Causal N\-gram Encoder*constructs multi\-scale local memory vectors from the input sequence using only the pretrained token embeddings, while the*Cosine\-Gated Memory Injector*measures their similarity to decoder hidden states and integrates the resulting memory update into the backbone through a residual connection\. Thus, the encoder specifies what local information is available as memory, and the injector determines when this information should influence the decoder\. Algorithm[1](https://arxiv.org/html/2605.16893#alg1)summarizes the overall procedure\. In the inference setting considered in this work, all backbone parameters remain frozen, and the only additional computation is induced by the current input sequence and a small set of predefined N\-gram sizes\.

### 3\.1Causal N\-gram Encoder

The first component of NGM is a Causal N\-gram Encoder, which converts the input prefix into multi\-scale local memory vectors using only the backbone model’s token embedding matrix\. Let the input token IDs be𝑿=\{x1,…,xT\}\\boldsymbol\{X\}=\\\{x\_\{1\},\\ldots,x\_\{T\}\\\}and the backbone token embedding matrix be𝑬∈ℝV×d\\boldsymbol\{E\}\\in\\mathbb\{R\}^\{V\\times d\}\. Token embeddings are𝒆t=𝑬​\[xt\]∈ℝd\\boldsymbol\{e\}\_\{t\}=\\boldsymbol\{E\}\[x\_\{t\}\]\\in\\mathbb\{R\}^\{d\}\. For eachn∈𝒩n\\in\\mathcal\{N\}\(e\.g\.,\{2,3\}\\\{2,3\\\}\), we first left\-pad the embedding sequence with\(n−1\)\(n\-1\)zero vectors to form𝒆~t\\tilde\{\\boldsymbol\{e\}\}\_\{t\}:

𝒆~t=\{𝟎if​2−n≤t≤0,𝒆tif​1≤t≤T\.\\tilde\{\\boldsymbol\{e\}\}\_\{t\}=\\begin\{cases\}\\boldsymbol\{0\}&\\text\{if \}2\-n\\leq t\\leq 0,\\\\ \\boldsymbol\{e\}\_\{t\}&\\text\{if \}1\\leq t\\leq T\.\\end\{cases\}\(1\)We then define a*causal*N\-gram representation at positionttby average pooling over a trailing window ofnntokens on the padded sequence:

𝒈t,n=1n​∑k=0n−1𝒆~t−k\.\\boldsymbol\{g\}\_\{t,n\}=\\frac\{1\}\{n\}\\sum\_\{k=0\}^\{n\-1\}\\tilde\{\\boldsymbol\{e\}\}\_\{t\-k\}\.\(2\)This uses a*bag\-of\-embeddings*approximation: the arithmetic mean can capture local patterns at different granularities without learning separate memory entries\. The resulting representation is intentionally order\-insensitive within the window and is not intended to recover full phrase semantics; rather, it provides a simple local summary that can be computed without additional parameters\. The left\-padding keeps the output length unchanged and ensures causality \(positionttdepends only on tokens≤t\\leq t\. For multiple window sizes, we stack the per\-size vectors into a matrix:

𝑮t=\[𝒈t,n\]n∈𝒩∈ℝ\|𝒩\|×d,\\boldsymbol\{G\}\_\{t\}=\\bigl\[\\boldsymbol\{g\}\_\{t,n\}\\bigr\]\_\{n\\in\\mathcal\{N\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{N\}\|\\times d\},\(3\)which is then consumed by the injector \(§[3\.2](https://arxiv.org/html/2605.16893#S3.SS2.SSS0.Px1)\)\. In implementation, the left\-padding and causal average pooling are realized withF\.padfollowed by 1D average pooling with kernel sizennand stride11, which is fully parallelizable\.

Algorithm 1NGM: N\-gram Memory \(*Causal N\-gram Encoder*\+*Cosine\-Gated Memory Injector*\)1:Token IDs

𝑿=\{x1,…,xT\}\\boldsymbol\{X\}=\\\{x\_\{1\},\\dots,x\_\{T\}\\\}; hidden states

𝑯l=\{𝒉tl\}t=1L\\boldsymbol\{H\}^\{l\}=\\\{\\boldsymbol\{h\}\_\{t\}^\{l\}\\\}\_\{t=1\}^\{L\}; backbone token embedding matrix

𝑬∈ℝV×d\\boldsymbol\{E\}\\in\\mathbb\{R\}^\{V\\times d\}; N\-gram sizes

𝒩\\mathcal\{N\}; output scale

λ\\lambda; boolean use\_relu\.

2:Updated hidden states

𝑯l′=\{𝒉tl′\}t=1L\\boldsymbol\{H\}^\{l^\{\\prime\}\}=\\\{\\boldsymbol\{h\}\_\{t\}^\{l^\{\\prime\}\}\\\}\_\{t=1\}^\{L\}\.

3:% — Causal N\-gram Encoder \(Eqs\. 1–4\) —

4:

𝑮←CausalNgramEncoder​\(𝑿,𝑬,𝒩\)T−L\+1:T\\boldsymbol\{G\}\\leftarrow\\textsc\{CausalNgramEncoder\}\(\\boldsymbol\{X\},\\boldsymbol\{E\},\\mathcal\{N\}\)\_\{T\-L\+1:T\}⊳\\triangleright𝑮∈ℝL×\|𝒩\|×d\\boldsymbol\{G\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times\|\\mathcal\{N\}\|\\times d\}, aligned to lastLLpositions

5:% — Cosine\-Gated Memory Injector —

6:

𝑯^l←L2Norm​\(𝑯l\)\\hat\{\\boldsymbol\{H\}\}^\{l\}\\leftarrow\\text\{L2Norm\}\(\\boldsymbol\{H\}^\{l\}\)⊳\\trianglerightRow\-wiseℓ2\\ell\_\{2\}normalization;𝑯^l∈ℝL×d\\hat\{\\boldsymbol\{H\}\}^\{l\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times d\}

7:

𝑮^←L2Norm​\(𝑮\)\\hat\{\\boldsymbol\{G\}\}\\leftarrow\\text\{L2Norm\}\(\\boldsymbol\{G\}\)⊳\\trianglerightNormalize alongdd;𝑮^∈ℝL×\|𝒩\|×d\\hat\{\\boldsymbol\{G\}\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times\|\\mathcal\{N\}\|\\times d\}

8:

St,n←∑jH^t,jl​G^t,n,jS\_\{t,n\}\\leftarrow\\sum\_\{j\}\\hat\{H\}^\{l\}\_\{t,j\}\\,\\hat\{G\}\_\{t,n,j\}⊳\\trianglerightPosition\-wise cosine similarity;𝑺∈ℝL×\|𝒩\|\\boldsymbol\{S\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times\|\\mathcal\{N\}\|\}

9:ifuse\_reluthen

10:

𝑺←max⁡\(𝟎,𝑺\)\\boldsymbol\{S\}\\leftarrow\\max\(\\boldsymbol\{0\},\\,\\boldsymbol\{S\}\)⊳\\trianglerightSuppress negatively aligned entries

11:endif

12:

Mt,j←∑nSt,n​Gt,n,jM\_\{t,j\}\\leftarrow\\sum\_\{n\}S\_\{t,n\}\\,G\_\{t,n,j\}⊳\\trianglerightGated weighted sum over N\-gram scales;𝑴∈ℝL×d\\boldsymbol\{M\}\\\!\\in\\\!\\mathbb\{R\}^\{L\\times d\}

13:

𝑯l′←𝑯l\+λ​𝑴\\boldsymbol\{H\}^\{l^\{\\prime\}\}\\leftarrow\\boldsymbol\{H\}^\{l\}\+\\lambda\\,\\boldsymbol\{M\}⊳\\trianglerightScaled residual injection

14:return

𝑯l′\\boldsymbol\{H\}^\{l^\{\\prime\}\}

### 3\.2Cosine\-Gated Memory Injector

The second component is a Cosine\-Gated Memory Injector, which measures compatibility between hidden states and the encoded local memory, then injects the resulting update through a residual path\. Given the decoder hidden state𝒉tl∈ℝd\\boldsymbol\{h\}\_\{t\}^\{l\}\\in\\mathbb\{R\}^\{d\}at layerlland positiontt, we compute a cosine similarity score with each𝒈t,n\\boldsymbol\{g\}\_\{t,n\}:

st,n=cos⁡\(𝒉tl,𝒈t,n\)=⟨𝒉tl,𝒈t,n⟩‖𝒉tl‖​‖𝒈t,n‖\.s\_\{t,n\}=\\cos\(\\boldsymbol\{h\}\_\{t\}^\{l\},\\boldsymbol\{g\}\_\{t,n\}\)=\\frac\{\\langle\\boldsymbol\{h\}\_\{t\}^\{l\},\\boldsymbol\{g\}\_\{t,n\}\\rangle\}\{\\\|\\boldsymbol\{h\}\_\{t\}^\{l\}\\\|\\\|\\boldsymbol\{g\}\_\{t,n\}\\\|\}\.\(4\)Optionally, we applyReLU\\mathrm\{ReLU\}to suppress negatively aligned updates:

s~t,n=max⁡\(0,st,n\)\.\\tilde\{s\}\_\{t,n\}=\\max\(0,s\_\{t,n\}\)\.\(5\)The aggregatedNN\-gram embeddings𝒈t,n\\boldsymbol\{g\}\_\{t,n\}serve as context\-local memory priors derived from the pretrained embedding space\. However, being constructed without additional training, these memory vectors should only be injected when they are compatible with the current decoder state\. Motivated by our empirical finding that aggregatedNN\-gram embeddings are geometrically aligned with Qwen3\-8B hidden states, we use the layer\-llhidden state𝒉tl\\boldsymbol\{h\}\_\{t\}^\{l\}as a context\-dependent query and measure its cosine similarity with each memory vector𝒈t,n\\boldsymbol\{g\}\_\{t,n\}\. This training\-free gate relies on the observed compatibility between the two representation spaces, enabling useful local memory signals to be selected and written back through a residual connection without learned projections, external retrieval, or additional parameters\.

#### Residual update and KV\-cache compatibility\.

Let𝒔~t=\[s~t,n\]n∈𝒩∈ℝ\|𝒩\|\\tilde\{\\boldsymbol\{s\}\}\_\{t\}=\[\\tilde\{s\}\_\{t,n\}\]\_\{n\\in\\mathcal\{N\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{N\}\|\}collect the gated scores\. We aggregate across N\-gram scales and inject the resulting memory signal through a residual connection:

𝒉tl′=𝒉tl\+λ​𝒔~t⊤​𝑮t,\\boldsymbol\{h\}\_\{t\}^\{l^\{\\prime\}\}=\\boldsymbol\{h\}\_\{t\}^\{l\}\+\\lambda\\,\\tilde\{\\boldsymbol\{s\}\}\_\{t\}^\{\\top\}\\boldsymbol\{G\}\_\{t\},\(6\)whereλ\\lambdais a scalar output scale that controls the magnitude of the injected update and𝑮t\\boldsymbol\{G\}\_\{t\}is defined in Eq\. \([3](https://arxiv.org/html/2605.16893#S3.E3)\)\. During autoregressive generation with KV cache, only the lastLLhidden states are computed at each decoding step\. We construct N\-gram embeddings from the full input ID prefix and slice the lastLLpositions to align with the currently available hidden states, preserving causal consistency under cached decoding\.

#### Complexity Analysis\.

We further analyze the computational overhead of NGM\. During the*prefill*phase, the full input sequence is processed in a single forward pass\. For sequence lengthTT, hidden dimensiondd, and N\-gram size set𝒩\\mathcal\{N\}, NGM adds causal pooling and position\-wise cosine scoring, both of which scale linearly withTTanddd\. Thus, the prefill complexity isO​\(T​\|𝒩\|​d\)O\(T\|\\mathcal\{N\}\|d\)\. During*autoregressive decoding*, the N\-gram representation at positionttdepends only on the most recentmax⁡\(𝒩\)\\max\(\\mathcal\{N\}\)token embeddings\. By caching these embeddings and updating𝒈t,n\\boldsymbol\{g\}\_\{t,n\}incrementally, a streaming implementation reduces the per\-step complexity toO​\(\|𝒩\|​d\)O\(\|\\mathcal\{N\}\|d\), which is independent of the prefix lengthTT\. Therefore, NGM incurs only linear overhead in prefill and constant overhead per decoding step\.

#### Layer integration\.

We insert the injector after the MLP block in selected decoder layers, specified by their layer IDs\. This placement keeps the self\-attention and feed\-forward parameters unchanged, while allowing the injected signal to act on contextualized hidden representations\. The insertion layer IDs are treated as hyperparameters rather than learnable components\. Following the layer\-selection strategy of Engram\[[8](https://arxiv.org/html/2605.16893#bib.bib1)\], we inject memory into a small set of early and middle layers, where residual\-alignment signals are empirically strongest\. We report the default layer placements for different models in Appendix Table[7](https://arxiv.org/html/2605.16893#A1.T7)\. In the inference setting considered in this work, all backbone parameters remain unchanged\. NGM introduces no new trainable weights and can be enabled or disabled at inference time for compatible checkpoints\.

## 4Experiments

### 4\.1Experimental setup

#### Setup\.

We evaluate NGM on the Qwen3 family\[[41](https://arxiv.org/html/2605.16893#bib.bib5)\], one of the most widely used open\-source model families, covering five model scales: 0\.6B, 1\.7B, 4B, 8B, and 14B\. We choose Qwen3 because it provides a consistent and publicly available series across a broad range of parameter sizes, making it well suited for controlled scaling analysis\. Other open\-source model families, such as Llama\[[18](https://arxiv.org/html/2605.16893#bib.bib12)\], DeepSeek\[[27](https://arxiv.org/html/2605.16893#bib.bib13)\], and Mistral\[[23](https://arxiv.org/html/2605.16893#bib.bib14)\], are less suitable for this particular setting because their publicly available checkpoints differ more substantially in release policy, model coverage, scale granularity, or evaluation comparability\. For each checkpoint, we compare the original model with the same model augmented by NGM, without updating the backbone weights\. Unless stated otherwise, we use𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\}and enable ReLU gating\. For each backbone, we keep a fixed output scale and a fixed set of insertion layers across tasks; these model\-specific settings are listed in Appendix Table[7](https://arxiv.org/html/2605.16893#A1.T7)\. All evaluations use EvalScope\[[37](https://arxiv.org/html/2605.16893#bib.bib30)\]; unless a benchmark requires task\-specific settings, the baseline and NGM share identical decoding parameters, with temperature=0\.7=0\.7, top\-p=0\.8p=0\.8, and top\-k=20k=20\.

#### Benchmarks\.

We report results on eight benchmarks spanning math, code, knowledge, and alignment: GSM8K\[[9](https://arxiv.org/html/2605.16893#bib.bib18)\], MATH500\[[21](https://arxiv.org/html/2605.16893#bib.bib19)\], HumanEval\[[6](https://arxiv.org/html/2605.16893#bib.bib20)\], LiveCodeBench v5\[[22](https://arxiv.org/html/2605.16893#bib.bib21)\], MMLU\-Redux\[[20](https://arxiv.org/html/2605.16893#bib.bib22),[17](https://arxiv.org/html/2605.16893#bib.bib23)\], GPQA\-Diamond\[[35](https://arxiv.org/html/2605.16893#bib.bib24)\], IFEval\[strict\-prompt;[43](https://arxiv.org/html/2605.16893#bib.bib25)\], and TruthfulQA\[MC2;[26](https://arxiv.org/html/2605.16893#bib.bib26)\]\. Unless noted otherwise, we follow standard benchmark protocols; for MMLU\-Redux, we use a context length of 4096\.

### 4\.2Main results

Table 1:Main results across five Qwen3 scales \(0\.6B–14B\)\. LCB = LiveCodeBench v5, MMLU\-R = MMLU\-Redux, GPQA = GPQA\-Diamond, IFEval = IFEval \(strict\-prompt\), and TQA = TruthfulQA \(MC2\)\. Each model is evaluated with and without NGM under identical decoding settings\. Best result within each model size isbold\.Table[1](https://arxiv.org/html/2605.16893#S4.T1)summarizes the main results\. Across the five tested model scales, NGM improves the average score in every case \(\+1\.2, \+0\.5, \+0\.6, \+0\.8, and \+0\.7 from 0\.6B to 14B\) while adding no new trainable parameters\. The clearest pattern appears on code benchmarks: LiveCodeBench improves at every tested scale, and HumanEval improves or matches the baseline at all scales\. Beyond code, the gains are positive but less uniform\. GSM8K improves at all tested scales, and GPQA improves at four of five scales, whereas MATH500 and MMLU\-Redux are more mixed\.

Alignment\-oriented tasks show a similar split\. TruthfulQA improves at most scales, while IFEval often degrades\. One plausible explanation is that NGM is training\-free and relies on a fixed, non\-learned residual injection\. As a result, the added local\-pattern signal can sometimes interfere with instruction\-sensitive control behavior instead of reinforcing it\. Even so, the broader gains are obtained without introducing additional trainable parameters or external knowledge, supporting the effectiveness of the core NGM mechanism itself\. Overall, these results are consistent with the view that NGM is most useful when short\-range pattern stability matters, rather than as a uniform improvement for all tasks\. As discussed in §[3\.2](https://arxiv.org/html/2605.16893#S3.SS2.SSS0.Px1), the additional overhead remains linear in prefix length and hidden size and does not change the asymptotic attention pattern\.

### 4\.3Extension to multimodal models

Table 2:Preliminary multimodal results on Qwen3\-VL\-2B\-Instruct\. MMBench = MMBench\_DEV\_EN\_V11\[[30](https://arxiv.org/html/2605.16893#bib.bib27)\], MMStar\[[5](https://arxiv.org/html/2605.16893#bib.bib28)\], OCRBench\[[31](https://arxiv.org/html/2605.16893#bib.bib29)\]\(scored out of 1000\), TQA = TruthfulQA \(MC2\)\[[26](https://arxiv.org/html/2605.16893#bib.bib26)\], and MMLU\-R = MMLU\-Redux\[[17](https://arxiv.org/html/2605.16893#bib.bib23)\]\. NGM is applied only to the language decoder; the visual encoder is unchanged\.To test whether NGM transfers beyond text\-only LLMs, we apply NGM to Qwen3\-VL\-2B\-Instruct\[[1](https://arxiv.org/html/2605.16893#bib.bib6)\], leaving the visual encoder and vision\-language fusion modules unchanged\. The N\-gram operates exclusively on text token embeddings; vision tokens are excluded from the sliding\-window pooling so that the local memory signal remains purely linguistic\. Using VLMEvalKit\[[13](https://arxiv.org/html/2605.16893#bib.bib47)\]under identical decoding settings, NGM improves or matches the baseline on all five multimodal and text benchmarks, with the largest gain on MMStar \(\+1\.53; Table[2](https://arxiv.org/html/2605.16893#S4.T2)\)\. This single\-scale result suggests that the same training\-free local\-memory mechanism can transfer to multimodal models without architectural changes, but we leave comprehensive multimodal evaluation to future work\.

### 4\.4Ablation studies

We study the sensitivity of NGM onQwen3\-8Bby varying one component at a time from the default configuration\. Unless noted otherwise, the default uses𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\},λ=0\.1\\lambda=0\.1, ReLU gating, stack fusion, and layers\{1,14\}\\\{1,14\\\}\(0\-based layer IDs\)\.

Table 3:Effect ofNN\-gram sizes on Qwen3\-8B\. HE = HumanEval, LCB = LiveCodeBench v5, MMLU\-R = MMLU\-Redux, GPQA = GPQA\-Diamond, IFEval = IFEval \(strict\-prompt\), TQA = TruthfulQA \(MC2\)\. Default:𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\}\. Other settings:λ=0\.1\\lambda=0\.1, ReLU, stack fusion, layers\{1,14\}\\\{1,14\\\}\(0\-based layer IDs\)\.#### NN\-gram sizes\.

Table[3](https://arxiv.org/html/2605.16893#S4.T3)compares different combinations ofNN\-gram window sizes\. Single\-scale variants help on some tasks, but multi\-scale settings perform better on average\. The default choice\{2,3\}\\\{2,3\\\}gives the strongest average result, while addingn=4n=4improves a few individual tasks without improving overall robustness\.

Table 4:Effect of ReLU gating on Qwen3\-8B\. Default: w/ ReLU\. Other settings:𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\},λ=0\.1\\lambda=0\.1, stack fusion, layers\{1,14\}\\\{1,14\\\}\(0\-based layer IDs\)\.
#### ReLU gating\.

Table[4](https://arxiv.org/html/2605.16893#S4.T4)compares ReLU\-filtered gating \(default\) with raw cosine gating\. ReLU is important for stable gains: removing it lowers the average score from 72\.17 to 70\.38, with the largest drop on LiveCodeBench\. This is consistent with the view that suppressing anti\-aligned updates helps avoid harmful residual injections\.

Table 5:Effect of fusion mode on Qwen3\-8B\. Default: stack\. Other settings:𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\},λ=0\.1\\lambda=0\.1, ReLU gating, layers\{1,14\}\\\{1,14\\\}\(0\-based layer IDs\)\.
#### Fusion mode: stack vs\. concat\.

In the defaultstackmode, each scalennhas its own cosine gate and the residual update is∑ns~t,n​𝒈t,n\\sum\_\{n\}\\tilde\{s\}\_\{t,n\}\\,\\boldsymbol\{g\}\_\{t,n\}\. Inconcatmode, per\-scale embeddings are concatenated into\[𝒈t,n\]n∈𝒩∈ℝ\|𝒩\|​d\[\\boldsymbol\{g\}\_\{t,n\}\]\_\{n\\in\\mathcal\{N\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{N\}\|d\}; the hidden state is tiled\|𝒩\|\|\\mathcal\{N\}\|times to match this dimensionality, a single scalar gate is computed via cosine similarity in the joint space, and the gate scales the mean embedding1\|𝒩\|​∑n𝒈t,n\\frac\{1\}\{\|\\mathcal\{N\}\|\}\\sum\_\{n\}\\boldsymbol\{g\}\_\{t,n\}\. Table[5](https://arxiv.org/html/2605.16893#S4.T5)shows that stack outperforms concat on average \(72\.17 vs\. 71\.07\): independent per\-scale gating is more flexible than collapsing all scales into one gating decision\.

Table 6:Effect of Compressed Tokenizer on Qwen3\-8B\. Default: w/o CompTok\. Other settings:𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\},λ=0\.1\\lambda=0\.1, ReLU gating, stack fusion, layers\{1,14\}\\\{1,14\\\}\(0\-based layer IDs\)\.
#### Compressed Tokenizer\.

Table[6](https://arxiv.org/html/2605.16893#S4.T6)tests whether applying the Engram\-style Compressed Tokenizer\[[8](https://arxiv.org/html/2605.16893#bib.bib1)\]—which maps subword tokens with the same normalized surface form to a shared ID before embedding lookup—benefits NGM’sNN\-gram construction\. It yields task\-specific gains, most notably on HumanEval, but does not improve the average score relative to the default\. We therefore keep the standard tokenizer as the default configuration\.

Taken together, these ablations indicate that multi\-scale construction and ReLU\-filtered gating are the most important contributors in the default setup, while stack fusion is a more reliable default than concat and the Compressed Tokenizer remains task\-dependent\.

### 4\.5Mechanistic analysis

We examine two mechanistic questions: whether the cosine gate reflects meaningful aligned structure rather than a generic embedding prior, and whether the resulting interactions are local in position\.

#### Interactions are predominantly local\.

We next examine the full cross\-position matrixcos⁡\(𝒉i,𝒈j\)\\cos\(\\boldsymbol\{h\}\_\{i\},\\boldsymbol\{g\}\_\{j\}\)on the same model \(Figure[3](https://arxiv.org/html/2605.16893#S4.F3)\)\. Across representative code, math, and knowledge samples, the diagonal mean consistently exceeds the off\-diagonal mean, with diagonal/off\-diagonal ratios of 1\.27×\\times–2\.42×\\timesat the injected layers\. The pattern is strongest for the knowledge sample and remains clear for code, indicating that the most useful memory signal is concentrated near the aligned position rather than uniformly distributed across the sequence\.

![Refer to caption](https://arxiv.org/html/2605.16893v1/x3.png)Figure 3:Cross\-position locality of NGM interactions in the default Qwen3\-8B\-NGM model\. Heatmaps show the average cross\-position cosine matrixcos⁡\(𝒉i,𝒈j\)\\cos\(\\boldsymbol\{h\}\_\{i\},\\boldsymbol\{g\}\_\{j\}\)at the two default injection layers for representative code, math, and knowledge samples\. The diagonal structure dominates, indicating that useful memory interactions are predominantly local\.
#### Implication for the default gate\.

Together, these results support the default design\. Since alignment remains well above shuffled and random controls, raw cosine similarity is informative without a learned projection\. Since the interaction pattern is diagonal\-dominant, the token\-wise gate captures most of the useful signal while preserving linear\-time cost\. This interpretation is consistent with the strongest gains in Table[1](https://arxiv.org/html/2605.16893#S4.T1), especially on tasks where short\-range pattern stability matters\. Combined with our intentionally restrictive evaluation setting—fixed backbone weights, no additional trainable parameters, and no external knowledge—this also yields a relatively controlled comparison in which the observed gains can be attributed more directly to the core NGM mechanism\. These mechanistic findings provide empirical support for the residual\-alignment intuition that motivates NGM, though they do not constitute a formal proof\.

### 4\.6Wall\-clock overhead

We measure prefill and decode latency for Qwen3\-8B with and without NGM on a single RTX 5090 \(batch size 1,bfloat16, prompt lengths 128–2048, 20 runs; Figure[4](https://arxiv.org/html/2605.16893#S4.F4)\)\. Overhead is reported as the relative latency increase over the original Qwen3\-8B baseline, i\.e\.,\(latencyN​G​M−latencybase\)/latencybase\(\\text\{latency\}\_\{NGM\{\}\}\-\\text\{latency\}\_\{\\text\{base\}\}\)/\\text\{latency\}\_\{\\text\{base\}\}\.

![Refer to caption](https://arxiv.org/html/2605.16893v1/x4.png)Figure 4:Prefill and per\-token decode latency for Qwen3\-8B vs\. Qwen3\-8B\-NGM on a single RTX 5090 \(mean±\\pmstd over 20 runs\)\. The gap widens at 2048 tokens because the current implementation recomputes N\-gram features over the full prefix\.For 256–1024\-token prompts, prefill overhead is 3\.4–7\.3% and decode overhead 1\.9–2\.3%; at 2048 tokens the figures rise to 16\.0% and 9\.9%, respectively\. These numbers reflect the released code path, which recomputes N\-gram features over the full prefix; a streaming cache would reduce the cost toO​\(\|𝒩\|​d\)O\(\|\\mathcal\{N\}\|d\)per step\.

## 5Conclusion

We presented NGM, a training\-free N\-gram memory module that constructs causal multi\-scaleNN\-gram representations from the backbone’s own token embeddings and injects them via a non\-parametric cosine gate, adding no trainable parameters\. Across Qwen3 models from 0\.6B to 14B, NGM improves average performance by \+0\.5 to \+1\.2 points, with the strongest gains on code generation\. The mixed task\-wise results point to the limits of a fixed injection rule; future work should explore lightly parameterized or task\-adaptive variants\.

## 6Limitations

NGM has three main limitations\. First, the causal N\-gram encoder uses a bag\-of\-embeddings approximation \(§[3](https://arxiv.org/html/2605.16893#S3)\), making it order\-insensitive; it may therefore mishandle order\-sensitive or non\-compositional phrases and inject misleading signals in such cases\. Second, the cosine gate and the model\-specific fixed scaleλ\\lambdaare heuristics rather than context\-adaptive learned components, and as reflected in the mixed task\-wise results, this training\-free injection rule does not suit all tasks or generation styles equally well\. Third, NGM reinforces short\-range regularities but does not retrieve external knowledge or directly address long\-range reasoning; it should therefore be viewed as complementary to long\-context mechanisms and retrieval\-based systems\[[3](https://arxiv.org/html/2605.16893#bib.bib16)\]\.

## References

- \[1\]\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§4\.3](https://arxiv.org/html/2605.16893#S4.SS3.p1.1)\.
- \[2\]P\. Bojanowski, E\. Grave, A\. Joulin, and T\. Mikolov\(2017\)Enriching word vectors with subword information\.Transactions of the association for computational linguistics5,pp\. 135–146\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[3\]S\. Borgeaud, A\. Mensch, J\. Hoffmann, T\. Cai, E\. Rutherford, K\. Millican, G\. B\. Van Den Driessche, J\. Lespiau, B\. Damoc, A\. Clark,et al\.\(2022\)Improving language models by retrieving from trillions of tokens\.InInternational conference on machine learning,pp\. 2206–2240\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7),[§6](https://arxiv.org/html/2605.16893#S6.p1.1)\.
- \[4\]T\. Brants, A\. Popat, P\. Xu, F\. J\. Och, and J\. Dean\(2007\)Large language models in machine translation\.InProceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning \(EMNLP\-CoNLL\),pp\. 858–867\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[5\]L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.Advances in Neural Information Processing Systems37,pp\. 27056–27087\.Cited by:[Table 2](https://arxiv.org/html/2605.16893#S4.T2)\.
- \[6\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[7\]S\. F\. Chen and J\. Goodman\(1999\)An empirical study of smoothing techniques for language modeling\.Computer Speech & Language13\(4\),pp\. 359–394\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[8\]X\. Cheng, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie, K\. Huang, X\. Yu, Z\. Hao, Y\. Li,et al\.\(2026\)Conditional memory via scalable lookup: a new axis of sparsity for large language models\.arXiv preprint arXiv:2601\.07372\.Cited by:[§A\.1](https://arxiv.org/html/2605.16893#A1.SS1.p2.1),[§1](https://arxiv.org/html/2605.16893#S1.p1.1),[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7),[§3\.2](https://arxiv.org/html/2605.16893#S3.SS2.SSS0.Px3.p1.1),[§4\.4](https://arxiv.org/html/2605.16893#S4.SS4.SSS0.Px4.p1.1)\.
- \[9\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[10\]M\. Constant, G\. Eryiğit, J\. Monti, L\. Van Der Plas, C\. Ramisch, M\. Rosner, and A\. Todirascu\(2017\)Survey: multiword expression processing: a survey\.Computational Linguistics43\(4\),pp\. 837–892\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[11\]A\. Y\. Din, T\. Karidi, L\. Choshen, and M\. Geva\(2024\)Jump to conclusions: short\-cutting transformers with linear transformations\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 9615–9625\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]N\. Ding, F\. Liu, K\. Kim, L\. Hao, K\. Lee, H\. Ko, and Y\. Tang\(2026\)MeKi: memory\-based expert knowledge injection for efficient llm scaling\.arXiv preprint arXiv:2602\.03359\.External Links:[Link](https://arxiv.org/abs/2602.03359)Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[13\]H\. Duan, J\. Yang, Y\. Qiao, X\. Fang, L\. Chen, Y\. Liu, X\. Dong, Y\. Zang, P\. Zhang, J\. Wang,et al\.\(2024\)Vlmevalkit: an open\-source toolkit for evaluating large multi\-modality models\.InProceedings of the 32nd ACM International Conference on Multimedia,pp\. 11198–11201\.Cited by:[§4\.3](https://arxiv.org/html/2605.16893#S4.SS3.p1.1)\.
- \[14\]N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread1\(1\),pp\. 12\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]B\. Erman\(2000\)The idiom principle and the open choice principle\.Text\-Interdisciplinary Journal for the Study of Discourse\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[16\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[17\]A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani,et al\.\(2025\)Are we done with mmlu?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5069–5096\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2605.16893#S4.T2)\.
- \[18\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px1.p1.4)\.
- \[19\]K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang\(2020\)Retrieval augmented language model pre\-training\.InInternational conference on machine learning,pp\. 3929–3938\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[20\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[21\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[22\]N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2024\)Livecodebench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[23\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.ArXivabs/2310\.06825\.External Links:[Link](https://api.semanticscholar.org/CorpusID:263830494)Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px1.p1.4)\.
- \[24\]U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis\(2019\)Generalization through memorization: nearest neighbor language models\.arXiv preprint arXiv:1911\.00172\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[25\]R\. Kneser and H\. Ney\(1995\)Improved backing\-off for m\-gram language modeling\.In1995 international conference on acoustics, speech, and signal processing,Vol\.1,pp\. 181–184\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[26\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)Truthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 3214–3252\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2605.16893#S4.T2)\.
- \[27\]A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px1.p1.4)\.
- \[28\]H\. Liu, J\. Zhang, C\. Wang, X\. Hu, L\. Lyu, J\. Sun, X\. Yang, B\. Wang, F\. Li, Y\. Qian, L\. Si, Y\. Sun, R\. Li, P\. Pei, Y\. Xie, and X\. Cai\(2026\)Scaling embeddings outperforms scaling experts in language models\.ArXivabs/2601\.21204\.External Links:[Link](https://api.semanticscholar.org/CorpusID:285140484)Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[29\]J\. Liu, S\. Min, L\. Zettlemoyer, Y\. Choi, and H\. Hajishirzi\(2024\)Infini\-gram: scaling unbounded n\-gram language models to a trillion tokens\.arXiv preprint arXiv:2401\.17377\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[30\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[Table 2](https://arxiv.org/html/2605.16893#S4.T2)\.
- \[31\]Y\. Liu, Z\. Li, M\. Huang, B\. Yang, W\. Yu, C\. Li, X\. Yin, C\. Liu, L\. Jin, and X\. Bai\(2024\)Ocrbench: on the hidden mystery of ocr in large multimodal models\.Science China Information Sciences67\(12\),pp\. 220102\.Cited by:[Table 2](https://arxiv.org/html/2605.16893#S4.T2)\.
- \[32\]G\. Neubig and C\. Dyer\(2016\)Generalizing and hybridizing count\-based and neural language models\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1163–1172\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[33\]T\. Nguyen\(2024\)Understanding transformers via n\-gram statistics\.Advances in neural information processing systems37,pp\. 98049–98082\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[34\]nostalgebraist\(2020\)Interpreting GPT: the logit lens\.Note:[https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px2.p1.1)\.
- \[35\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2024\)Gpqa: a graduate\-level google\-proof q&a benchmark\.InFirst conference on language modeling,Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.
- \[36\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[37\]M\. Team\(2024\)EvalScope: evaluation framework for large models\.External Links:[Link](https://github.com/modelscope/evalscope)Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px1.p1.4)\.
- \[38\]A\. Tseng and C\. De Sa\(2026\)L3: large lookup layers\.arXiv preprint arXiv:2601\.21461\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[39\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1)\.
- \[40\]Y\. Wu, M\. N\. Rabe, D\. Hutchins, and C\. Szegedy\(2022\)Memorizing transformers\.arXiv preprint arXiv:2203\.08913\.Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p1.1),[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[41\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px1.p1.4)\.
- \[42\]D\. Yu, E\. Cohen, B\. Ghazi, Y\. Huang, P\. Kamath, R\. Kumar, D\. Liu, and C\. Zhang\(2025\)Scaling embedding layers in language models\.ArXivabs/2502\.01637\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276106917)Cited by:[§1](https://arxiv.org/html/2605.16893#S1.p2.2),[§2](https://arxiv.org/html/2605.16893#S2.SS0.SSS0.Px1.p1.7)\.
- \[43\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§4\.1](https://arxiv.org/html/2605.16893#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix ANGM implementation

Listing[1](https://arxiv.org/html/2605.16893#LST1)gives a simplified PyTorch implementation of NGM\.

defngm\_forward\(hidden\_states,input\_ids,embed\_matrix,

ngram\_sizes,output\_scale,use\_relu\):

"""

hidden\_states:\[B,L,D\]\(layer\-lhiddenstates\)

input\_ids:\[B,T\]\(fullinputtokenids\)

embed\_matrix:Embedding\(tokenembeddingsfrombackbonemodel\)

"""

B,L,D=hidden\_states\.shape

T=input\_ids\.shape\[1\]

token\_emb=embed\_matrix\(input\_ids\)

ngram\_list=\[\]

forninngram\_sizes:

padded=F\.pad\(token\_emb,\(0,0,n\-1,0\)\)

pooled=F\.avg\_pool1d\(

padded\.transpose\(1,2\),kernel\_size=n,stride=1

\)\.transpose\(1,2\)

ngram\_list\.append\(pooled\)

ngram\_emb=torch\.stack\(ngram\_list,dim=2\)

ngram\_emb=ngram\_emb\[:,\-L:,:,:\]

h\_norm=F\.normalize\(hidden\_states,dim=\-1\)

g\_norm=F\.normalize\(ngram\_emb,dim=\-1\)

sim=torch\.einsum\(’bld,blnd\-\>bln’,h\_norm,g\_norm\)

ifuse\_relu:

sim=F\.relu\(sim\)

out=torch\.einsum\(’bln,blnd\-\>bld’,sim,ngram\_emb\)

returnhidden\_states\+output\_scale\*out

Listing 1:NGM core implementation \(simplified\)\.### A\.1Model\-specific NGM settings

Unless noted otherwise, all reported models use𝒩=\{2,3\}\\mathcal\{N\}=\\\{2,3\\\}and ReLU gating\. Table[7](https://arxiv.org/html/2605.16893#A1.T7)lists the model\-specific insertion layers, backbone depth, and output scales used for the converted Qwen3\-NGM checkpoints\. We report inserted decoder layers in 1\-based numbering for readability\.

Table 7:Model\-specific NGM settings for the converted Qwen3 checkpoints used in the experiments\.Our layer choices are heuristic but not arbitrary\. They were informed by Engram’s layer\-sensitivity analysis\[[8](https://arxiv.org/html/2605.16893#bib.bib1)\], which argues that memory injection should balance two considerations: placing memory early enough to offload local pattern reconstruction before the backbone expends much depth, and placing memory late enough that the hidden state used for gating is already meaningfully contextualized\. Guided by this trade\-off, we place one NGM module near the bottom of the network \(the 2nd decoder layer in all models\) and a second module at a deeper layer\. For Qwen3 models up to 8B parameters, we use the 15th decoder layer; for the deeper 14B model, we move the second insertion to the 20th decoder layer\. These settings were chosen with reference to Engram, the depth of each Qwen3 backbone, and the default configurations used in our experiments, and should be interpreted as practical defaults rather than universally optimal placements\.

## Appendix BCase studies: NGM vs\. base model on GSM8K

We present three representative examples from the GSM8K benchmark where Qwen3\-8B\-NGM produces the correct answer while the base Qwen3\-8B model fails\. Each case highlights a different failure mode that NGM helps mitigate\.

Case 1: Arithmetic error correctionQuestion\.In a candy machine, there are 22 more than four times the number of pink gumballs as there are blue gumballs\. If there are 12 blue gumballs, how many pink ones are there?Gold answer:70Base model \(Qwen3\-8B\) ✗ The number of pink gumballs is 22 more than four times the number of blue gumballs\.Given that there are 12 blue gumballs, we calculate:4×12=𝟐𝟖4\\times 12=\\boldsymbol\{28\}\(arithmetic error\) 28\+22=5028\+22=50 ANSWER: 50Qwen3\-8B\-NGM ✓ The number of pink gumballs is 22 more than four times the number of blue gumballs\.There are 12 blue gumballs\.4×12=𝟒𝟖4\\times 12=\\boldsymbol\{48\}\(correct\) 48\+22=7048\+22=70 ANSWER: 70Analysis\.The base model produces an incorrect intermediate product \(4×12=284\\times 12=28\), causing the final answer to be wrong\. NGM maintains correct arithmetic in the short\-range computation chain\.

Case 2: Semantic misinterpretationQuestion\.Finn watches 11 tadpoles swimming in the pond\. Suddenly he sees 6 of them come out of hiding from under a lily pad, then he sees 2 of them hide under a rock\. How many tadpoles can Finn see in the pond now?Gold answer:15Base model \(Qwen3\-8B\) ✗ Finn initially sees 11 tadpoles\.Then, 6 come out of hiding from under a lily pad\.After that, 2 hide under a rock\.Ignores the 6 newcomers: 11−2=911\-2=9 ANSWER: 9Qwen3\-8B\-NGM ✓ Finn initially watches 11 tadpoles\.Then, 6 come out of hiding→\\rightarrowvisible count increases by 6\.Next, 2 hide under a rock→\\rightarrowvisible count decreases by 2\.11\+6−2=1511\+6\-2=15ANSWER: 15Analysis\.The base model misinterprets “come out of hiding” and ignores the additional 6 tadpoles, computing only11−211\-2\. NGM correctly accumulates all quantity changes mentioned in the prompt\.

Case 3: Reading comprehension errorQuestion\.Three friends spent $20\.25 on 3 tickets to the fair\. They also spent $4\.50 less on food than on the tickets\. They also went on 2 different rides which cost $33 for each ride\. If they agreed to split all the costs evenly, how much did each of them pay?Gold answer:34Base model \(Qwen3\-8B\) ✗ Misreads $20\.25 as per\-ticket price: Tickets:3×$​20\.25=$​60\.753\\times\\mathdollar 20\.25=\\mathdollar 60\.75Food:$​60\.75−$​4\.50=$​56\.25\\mathdollar 60\.75\-\\mathdollar 4\.50=\\mathdollar 56\.25Rides:2×$​33=$​662\\times\\mathdollar 33=\\mathdollar 66Total:$​60\.75\+$​56\.25\+$​66=$​183\\mathdollar 60\.75\+\\mathdollar 56\.25\+\\mathdollar 66=\\mathdollar 183Each:$​183÷3=$​61\\mathdollar 183\\div 3=\\mathdollar 61ANSWER: 61Qwen3\-8B\-NGM ✓ Tickets: $20\.25 \(total for 3\)Food:$​20\.25−$​4\.50=$​15\.75\\mathdollar 20\.25\-\\mathdollar 4\.50=\\mathdollar 15\.75Rides:2×$​33=$​662\\times\\mathdollar 33=\\mathdollar 66Total:$​20\.25\+$​15\.75\+$​66=$​102\\mathdollar 20\.25\+\\mathdollar 15\.75\+\\mathdollar 66=\\mathdollar 102Each:$​102÷3=$​34\\mathdollar 102\\div 3=\\mathdollar 34ANSWER: 34Analysis\.The base model misreads “spent $20\.25 on 3 tickets” as $20\.25*per ticket*, inflating all downstream totals\. NGM correctly interprets $20\.25 as the total ticket expenditure, preserving local textual coherence across the reasoning chain\.