Tokenizer-Agnostic Engram Module

# Tokenizer-Agnostic Engram Module
Source: [https://arxiv.org/html/2607.29065](https://arxiv.org/html/2607.29065)
###### Abstract

Deepseek’s Engram, a conditional memory module, was introduced to trade\-off storage versus reasoning in large language models\. However, the module relies on token\-levelNN\-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch\. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers\. Instead of modelling disjointNN\-gram spaces, we treatNN\-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens\. We replace the XOR\-based hashing with the general polynomial hashing with a joint embedding space acrossNN\. This work investigates the possible trade\-offs and shows that this simple substitution produces comparable performance and achieves tokenizer\-agnosticism: hash equivalence for byte\-equivalent token sequences\.

Code—https://github\.com/jararap/polyhash\-engram

## Introduction

#### DeepSeek’s Engram Module\.

Their influential work\(Chenget al\.[2026](https://arxiv.org/html/2607.29065#bib.bib31)\)proposes using token\-levelNN\-gram conditional embedding lookup as an efficient parameter store\. Each token has an additional vector input, depending on the previousNN\-1 token sequence111Pad tokens are placeholders for the tokens at the start\., with each value acting as a hash key to the Engram embedding table\. For eachn∈\(1,N\]n\\in\(1,N\], multiple embedding vectors \(heads\) are sampled from the corresponding embedding table, i\.e\., eachnn\-gram space is disjoint\. These concatenated Engram embeddings alongNNserve as the key/value input to the scaled dot product attention \(SPDA\)\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.29065#bib.bib15)\), with the query being the residual of the previous model block\. The context\-aware SPDA gates the retrieved embeddings, before the 1D convolution operation along the token\-level axis and adding the result back to the residual stream\.

#### Tokenizer lock in\.

Tokenizers may share the same algorithm, e\.g\., Byte\-Pair Encoding \(BPE\)\(Gage[1994](https://arxiv.org/html/2607.29065#bib.bib25); Sennrichet al\.[2016](https://arxiv.org/html/2607.29065#bib.bib28)\)or Unigram\(Kudo and Richardson[2018](https://arxiv.org/html/2607.29065#bib.bib27); Kudo[2018](https://arxiv.org/html/2607.29065#bib.bib26)\), but may differ in their vocabulary set and size, resulting in different token sequence inputs\. Each token has a corresponding initial hash value with the final hash key\(s\) being an aggregation of these token hashes in a rolling XOR\-wise manner\. This causes a tight coupling between the Engram embeddings to the tokenizer used\. To share Engram embeddings across different models, they must use the same tokenizer\. Model performance scales with tokenizer vocabulary size\(Taoet al\.[2024](https://arxiv.org/html/2607.29065#bib.bib33); Takaseet al\.[2025](https://arxiv.org/html/2607.29065#bib.bib34)\), so being able to adjust the tokenizer, with respect to model size or architecture, is an important consideration\. For models with fewer parameters, a fixed tokenizer requirement hurts the flexibility to control the ratio of embedding to feedforward parameters\.

#### NN\-gram modelling an illusion?

NN\-grams are widely studied and used to evaluate and model local information\(Brantset al\.[2007](https://arxiv.org/html/2607.29065#bib.bib37); Bucket al\.[2014](https://arxiv.org/html/2607.29065#bib.bib36); Liuet al\.[2024](https://arxiv.org/html/2607.29065#bib.bib38); Nguyen[2024](https://arxiv.org/html/2607.29065#bib.bib39); Merrillet al\.[2024](https://arxiv.org/html/2607.29065#bib.bib35)\)\. We argue that what is crucial is the underlying byte sequence of theNN\-grams rather than modelling specificnn\-gram tokens\. Consider this case where our two tokenizers,TAT\_\{A\}andTBT\_\{B\}, tokenize stringSSinto a22\-gram and33\-gram respectively\. IfSSis informative, ideally, it should not matter in whichnn\-gram embedding space this information is stored\. We viewNN\-gram as a sampling method to shortlist potential byte sequences, from an exponential number of permutations, for the model to learn and determine its usefulness\.

#### Tokenizer Agnosticism\.

Hence, our goal is to modify the Engram Module such that it can be \(re\)used regardless of tokenizer choice\. In this work, our contributions are as follow:

1. 1\.Define the objective of Tokenizer Agnosticsm with respect to the Engram module, i\.e\., its hashing component\.
2. 2\.Propose a straightforward replacement of XOR hashing with the general polynomial hashing, detailing its advantages and maintaining similar algorithmic efficiency\.
3. 3\.Investigate the potential trade\-offs of modelling in a jointNN\-gram embedding space\. From our experiments, we obtain comparable results that suggest no downsides\.222We provide our training and data processing code, built on top of Lightning and HuggingFace, in the Code and Data Supplement\.
4. 4\.We train a model with a different tokenizer on pretrained Engram embeddings for cross\-tokenizer transfer, empirically showing that tokenizer agnosticism can be achieved\.

Example Text: “This is Genghis Khan”TokenizerTokensMistral’<s\>’’\_This’’\_is’’\_Gen’’gh’’is’’\_Khan’’</s\>’SmolLM2’This’’Ġis’’ĠGen’’gh’’is’’ĠKhan’’<\|endoftext\|\>’cl100k\_base’This’’Ġis’’ĠG’’eng’’his’’ĠKhan’’<\|endoftext\|\>’Table 1:A simple example showing the possible differences from using different tokenizers: \(i\) order not guaranteed due to possible shifts \(seeMistral/SmolLM2\); \(ii\)NN\-grams with different partitions \(see ’Genghis’ inSmolLM2/cl100k\_base\); \(iii\) tokenizer specific characteristics, e\.g\., space prefix, tokenizer indices, etc\. For this example, we use the text representation of the tokens\. These confounding factors complicates the mapping between the sameNN\-grams from different tokenizers\.

## Related Works

#### Tokenization\.

Besides common tokenization methods such as BPE and Unigram\. SuperBPE\(Liuet al\.[2025](https://arxiv.org/html/2607.29065#bib.bib30)\)and BoundlessBPE\(Schmidtet al\.[2025](https://arxiv.org/html/2607.29065#bib.bib29)\)are tokenization methods that include tokens across predefined delimiters, e\.g\., white spaces\. These additional tokens selected can be viewed as token\-levelNN\-grams\. TheseNN\-grams are preselected by the tokenizer algorithm, whereas for Engram module, the model learns whichNN\-grams are useful during training\.

#### Engram Module Adjacent\.

There are other alternative embedding modules/layers\(Google Team[2025](https://arxiv.org/html/2607.29065#bib.bib40); Tseng and Sa[2026](https://arxiv.org/html/2607.29065#bib.bib41); Sadhukhanet al\.[2026](https://arxiv.org/html/2607.29065#bib.bib42)\)whose mechanisms are different from Deepseek’s Engram\. Our work focuses on removing the latter’s tokenizer requirement\.Zhenget al\.\([2026](https://arxiv.org/html/2607.29065#bib.bib43)\)proposes Lngram which learns discrete symbols from the latent space for embedding lookup; and not via tokens\. It trades compute efficiency, as the lookup is reliant on the previous model block’s output for model performance\.

#### Hash Embeddings\.

First proposed bySvenstrupet al\.\([2017](https://arxiv.org/html/2607.29065#bib.bib48)\), it shares similarity with DeepSeek’s multihead embeddings and context\-aware gating\(Chenget al\.[2026](https://arxiv.org/html/2607.29065#bib.bib31)\)\. Our work also uses a shared embedding space, with the main difference being hashing from byte sequences to achieve the byte\-equivalence property\.

![Refer to caption](https://arxiv.org/html/2607.29065v1/x1.png)\(a\)Original implementation by DeepSeek\(Chenget al\.[2026](https://arxiv.org/html/2607.29065#bib.bib31)\)\.
![Refer to caption](https://arxiv.org/html/2607.29065v1/x2.png)\(b\)Ours: we hash from bytes, include11\-gram and use shared joint embeddings\.

Figure 1:Visualization of the algorithmic and architectural changes between the original approach and our proposed approach\. For our example, we setN=3N=3, and we use a token sequenceT​\(S\)T\(S\)of three tokens with different byte lengths\.

## Hashing for Tokenizer\-Agnosticism

Extending the original tokenizer compression rules, e\.g\., such as capitalization and space prefix, we also account for special tokens and space prefixes between different tokenizers\. However, they are still likely to produce different token sequences\. Even if they share some token vocabulary, there is no guarantee that the token sequences will be in the same order or that they will be partitioned using the same tokens\.

#### Example\.

In Table[1](https://arxiv.org/html/2607.29065#Sx1.T1), we show \(i\) token sequences fromMistralandSmolLM2differ in token order, and \(ii\) token sequences fromSmolLM2andcl100k\_base, “Genghis” are partitioned differently\.

### Problem: Hashing Equivalence between Tokenizers

We illustrate the problem using two different toy BPE tokenizersTTand their indexed token sets, only processing strings containing characters ‘a’ and ‘b’\.

TA:\{0: ‘aa’, 1: ‘a’, 2: ‘b’\}T\_\{A\}:\\\{\\text\{0: \`aa', 1: \`a', 2: \`b'\}\\\}TB:\{0: ‘a’, 1: ‘b’\}T\_\{B\}:\\\{\\text\{0: \`a', 1: \`b'\}\\\}
Given a string textS=“aabb”S=\\text\{\`\`aabb''\}, the respective tokenizers will produce the following string token sequence\.

TA​\(S\):\(‘aa’, ‘b’, ‘b’\)T\_\{A\}\(S\):\(\\text\{\`aa', \`b', \`b'\}\)TB​\(S\):\(‘a’, ‘a’, ‘b’, ’b’\)T\_\{B\}\(S\):\(\\text\{\`a', \`a', \`b', 'b'\}\)
Likewise with actual tokenizers, it is not guaranteed that two different tokenizers will produce the same tokenized output\. We define∥T\(S\)\\\|T\(S\)as the in\-sequence concatenation of bytes fromT​\(S\)T\(S\)\. It is evident thatSSis recovered from∥TA\(S\)\\\|T\_\{A\}\(S\)and∥TB\(S\)\\\|T\_\{B\}\(S\), therefore they arebyte equivalent\.

∥TA\(S\)≡∥TB\(S\)\\\|T\_\{A\}\(S\)\\equiv\\\|T\_\{B\}\(S\)\(1\)A naïve approach to map byte\-equivalentNN\-grams would involve comparing their bytes\. Byte equivalence ensures that we can map different token sequences of the same bytes to the same embedding index\.

H​a​s​h​\(TA​\(S\)\)≡H​a​s​h​\(TB​\(S\)\)\.Hash\(T\_\{A\}\(S\)\)\\equiv Hash\(T\_\{B\}\(S\)\)\.
Notice that for this exampleSS, the 3\-gram ofTAT\_\{A\}is byte\-equivalent to the 4\-gram ofTBT\_\{B\}\. To achieve tokenizer\-agnoticism, it is clear that we require byte\-level information, and not only rely on token\-level information, to ensure equivalence ofNN\-grams from different tokenizers\. However, this requirement itself presents further challenges:

1. 1\.Time\. Comparing bytes instead of integer tokens results in a more expensive computation as\|S\|≥\|T​\(S\)\|\|S\|\\geq\|T\(S\)\|\.
2. 2\.Space\. Naïvely storing permutations will not work for large vocabulary size as it requiresO​\(\|T\|N\)O\(\|T\|^\{N\}\)space\.

An efficient method will involve addressing these concerns, in addition to ensuring byte\-equivalentNN\-grams, using common strategies such as:

1. i\.Caching hashes for each integer token to reduce repeated computations\.
2. ii\.On\-demand calculation of finalNN\-gram hashes from the cached integer token hashes\.

#### Objective\.

For any token sequences up to lengthNN, we expecthash equivalencebetween the different integer token sequences, from different tokenizers, when there is byte equivalence:

H​a​s​h​\(TA​\(S\)\)≡H​a​s​h​\(TB​\(S\)\)\|Eq\.[1](https://arxiv.org/html/2607.29065#Sx3.E1)\.Hash\(T\_\{A\}\(S\)\)\\equiv Hash\(T\_\{B\}\(S\)\)\|\\text\{\\;Eq\. \\ref\{eq:byte\_equiv\}\}\.\(2\)Note that hash collisions might occur, i\.e\., there is hash equivalence even when Eq\.[1](https://arxiv.org/html/2607.29065#Sx3.E1)is not true, simply due to chance\.

### XOR\-hashing

In the original implementation,NN\-gram indices were derived by applying XOR to token hashes \(see Fig\.[1\(a\)](https://arxiv.org/html/2607.29065#Sx2.F1.sf1)\)\. Given a mapping of token indices to hashhhand hyperparametersb​a​s​ebaseandMM\(allocated Engram embedding table size\),

H​a​s​hx​\(T​\(S\)\)=\[⨁i=0\|T​\(S\)\|−1h​\(T​\(S\)i\)⋅b​a​s​ei\]modM\.Hash\_\{x\}\(T\(S\)\)=\[\\bigoplus^\{\|T\(S\)\|\-1\}\_\{i=0\}h\(T\(S\)\_\{i\}\)\\cdot base\_\{i\}\]\\mod M\.This is insufficient to achieve hash equivalence for byte\-equivalentNN\-grams\.

#### Counterexample 1\.

Two byte\-equivalentNN\-grams from different tokenizers might differ in the partitions\. GivenS1=\(‘​a​b′,‘​a′\)S\_\{1\}=\(\`ab^\{\\prime\},\`a^\{\\prime\}\)andS2=\(‘​a′,‘​b​a′\)S\_\{2\}=\(\`a^\{\\prime\},\`ba^\{\\prime\}\), with different token hasheshhandMMomitted, unless collision by chance,

h​\(‘​a​b′\)⋅b​a​s​e0⊕h​\(‘​a′\)⋅b​a​s​e1≢h​\(‘​a′\)⋅b​a​s​e0⊕h​\(‘​b​a′\)⋅b​a​s​e1\.h\(\`ab^\{\\prime\}\)\\cdot base\_\{0\}\\oplus h\(\`a^\{\\prime\}\)\\cdot base\_\{1\}\\not\\equiv h\(\`a^\{\\prime\}\)\\cdot base\_\{0\}\\oplus h\(\`ba^\{\\prime\}\)\\cdot base\_\{1\}\.

#### Counterexample 2\.

Two byte\-equivalentNN\-grams differ inNN\. GivenS1=\(‘​a​b​a′\)S\_\{1\}=\(\`aba^\{\\prime\}\)andS2=\(‘​a′,‘​b​a′\)S\_\{2\}=\(\`a^\{\\prime\},\`ba^\{\\prime\}\), with different token hasheshhandMMomitted, unless collision by chance,

h\(′aba′\)⋅base0≢h\(‘a′\)⋅base0⊕h\(‘ba′\)⋅base1\.h\(^\{\\prime\}aba^\{\\prime\}\)\\cdot base\_\{0\}\\not\\equiv h\(\`a^\{\\prime\}\)\\cdot base\_\{0\}\\oplus h\(\`ba^\{\\prime\}\)\\cdot base\_\{1\}\.For both examples, we get different hashes for byte\-equivalentNN\-grams\. A simple fix could involve removing the positionalb​a​s​ebasehyperparameters\. However, XOR’s commutative property increases hash collisions:

1. 1\.Information destruction\. GivenS1=\(‘​a′,‘​b′,‘​a′\)S\_\{1\}=\(\`a^\{\\prime\},\`b^\{\\prime\},\`a^\{\\prime\}\),S2=\(‘​b′,‘​a′,‘​a′\)S\_\{2\}=\(\`b^\{\\prime\},\`a^\{\\prime\},\`a^\{\\prime\}\), and a byte\-hash map functionhh\. Ah​\(‘​a′\)⊕h​\(‘​a′\)=0h\(\`a^\{\\prime\}\)\\oplus h\(\`a^\{\\prime\}\)=0, which makes it a hash of odd\-count unique tokens or bytes,H​a​s​h​\(S1\)≡H​a​s​h​\(S2\)Hash\(S\_\{1\}\)\\equiv Hash\(S\_\{2\}\)whenS1≢S2S\_\{1\}\\not\\equiv S\_\{2\}\.
2. 2\.Lack of position information\. GivenS3=\(‘​a′,‘​b′,‘​c′\)S\_\{3\}=\(\`a^\{\\prime\},\`b^\{\\prime\},\`c^\{\\prime\}\),S4=\(‘​c′,‘​a′,‘​b′\)S\_\{4\}=\(\`c^\{\\prime\},\`a^\{\\prime\},\`b^\{\\prime\}\), the order independence results inH​a​s​h​\(S3\)≡H​a​s​h​\(S4\)Hash\(S\_\{3\}\)\\equiv Hash\(S\_\{4\}\)whenS3≢S4S\_\{3\}\\not\\equiv S\_\{4\}, affecting the byte anagrams produced by the previous scenario\.

Another possible approach involves mapping the tokens between two tokenizers\. However, it is unclear how these non\-one\-to\-one hash embeddings will aggregate or disentangle during training\. Such methods are likely to require additional complex mechanics\.

### Polynomial\-hashing as an Efficient Alternative

To achieve hash equivalence for byte\-equivalent sequences, we need to accumulate hash information\. Polynomial\-hashing\(Carter and Wegman[1979](https://arxiv.org/html/2607.29065#bib.bib45); Bhattacharyyaet al\.[2025](https://arxiv.org/html/2607.29065#bib.bib46)\)is a common hashing approach that fits this criterion\. Given stringSS, byte\-hash map functionhh, and hyperparameterb​a​s​ebaseandMM\(Engram embedding table size\),

H​a​s​hp​\(S\)=\[∑i=0\|S\|−1h​\(si\)⋅b​a​s​e\|S\|−i\]modM\.Hash\_\{p\}\(S\)=\[\\sum^\{\|S\|\-1\}\_\{i=0\}h\(s\_\{i\}\)\\cdot base^\{\|S\|\-i\}\]\\mod M\.\(3\)Eq\.[3](https://arxiv.org/html/2607.29065#Sx3.E3)can be rewritten as an equivalent streaming variant,

H​a​s​hp​\(S\)=\[…\(h\(s0\)⋅base\+h\(s1\)\)⋅base…\+h\(s\|S\|−1\)\)⋅base\]modM\.\\displaystyle\\begin\{split\}Hash\_\{p\}\(S\)&=\[\\dots\(h\(s\_\{0\}\)\\cdot base\+h\(s\_\{1\}\)\)\\cdot base\\dots\\\\ &\+h\(s\_\{\|S\|\-1\}\)\)\\cdot base\]\\mod M\.\\end\{split\}\(4\)However, as previously described, we have to cache the token hashes to efficiently calculate the finalNN\-gram hash\. We can do so by caching the hash of each tokent∈Tt\\in T,C​\(t\)C\(t\)\(Eq\.[5](https://arxiv.org/html/2607.29065#Sx3.E5)\), andb​a​s​ebaseexponents,B​\(p​o​w​e​r\)B\(power\)\(Eq\.[6](https://arxiv.org/html/2607.29065#Sx3.E6)\)\.

C​\(t\)=H​a​s​hp​\(t\)C\(t\)=Hash\_\{p\}\(t\)\(5\)B​\(p​o​w​e​r\)=b​a​s​ep​o​w​e​rB\(power\)=base^\{power\}\(6\)We can then modify Eq\.[3](https://arxiv.org/html/2607.29065#Sx3.E3)to use Eq\.[5](https://arxiv.org/html/2607.29065#Sx3.E5)and Eq\.[6](https://arxiv.org/html/2607.29065#Sx3.E6),

H​a​s​hp​\(T​\(S\)\)=\[∑i=0\|T​\(S\)\|−1C\(T\(S\)i\)⋅B\(∑j=i\+1\|T​\(S\)\|−1\|T\(S\)j\|\)\]modM\.\\displaystyle\\begin\{split\}Hash\_\{p\}\(T\(S\)\)=&\[\\sum\_\{i=0\}^\{\|T\(S\)\|\-1\}C\(T\(S\)\_\{i\}\)\\\\ &\\cdot B\(\\sum\_\{j=i\+1\}^\{\|T\(S\)\|\-1\}\|T\(S\)\_\{j\}\|\)\]\\mod M\.\\end\{split\}\(7\)The equivalent streaming variant of Eq\.[7](https://arxiv.org/html/2607.29065#Sx3.E7)is more convenient as it removes the requirement to track remaining length,

Hashp\(T\(S\)\)=\[…\(C\(T\(S\)0\)⋅B\(\|T\(S\)1\|\)\+C\(T\(S\)1\)\)⋅B\(\|T\(S\)2\|\)…\+C\(T\(S\)\|T​\(S\)\|−1\)\)⋅B\(0\)\]modM\.\\displaystyle\\begin\{split\}Hash\_\{p\}\(&T\(S\)\)=\[\\dots\(C\(T\(S\)\_\{0\}\)\\cdot B\(\|T\(S\)\_\{1\}\|\)\\\\ &\+C\(T\(S\)\_\{1\}\)\)\\cdot B\(\|T\(S\)\_\{2\}\|\)\\dots\\\\ &\+C\(T\(S\)\_\{\|T\(S\)\|\-1\}\)\)\\cdot B\(0\)\]\\mod M\.\\end\{split\}\(8\)After initializing the cache, computing the polynomial hash of a token sequenceT​\(S\)T\(S\)isO​\(\|T​\(S\)\|⋅N\)O\(\|T\(S\)\|\\cdot N\)and can be computed using matrix operations, achieving the same complexity as original XOR\-hash routine\.

#### Example\.

Given stringS=‘​‘​a​b​c​d′′S=\`\`abcd^\{\\prime\\prime\}, where its token sequenceT​\(S\)=\(‘​‘​a​b′′,‘​‘​c​d′′\)T\(S\)=\(\`\`ab^\{\\prime\\prime\},\`\`cd^\{\\prime\\prime\}\), following Eq\.[4](https://arxiv.org/html/2607.29065#Sx3.E4), we get

Hashp\(S\)=\[\(\(\(h\(‘a′\)⋅base\+h\(‘b′\)\)⋅base\+h\(‘c′\)\)⋅base\+h\(‘d′\)\)⋅base\]modM\.\\displaystyle\\begin\{split\}Hash\_\{p\}\(S\)=\[\(\(\(h\(\`a^\{\\prime\}\)\\cdot base\+h\(\`b^\{\\prime\}\)\)\\cdot base\\\\ \+h\(\`c^\{\\prime\}\)\)\\cdot base\+h\(\`d^\{\\prime\}\)\)\\cdot base\]\\mod M\.\\end\{split\}\(9\)The following cached values for its tokensT​\(S\)T\(S\)are

C​\(‘​‘​a​b′′\)\\displaystyle C\(\`\`ab^\{\\prime\\prime\}\)=\(h​\(‘​a′\)⋅b​a​s​e\+h​\(‘​b′\)\)⋅b​a​s​e,\\displaystyle=\(h\(\`a^\{\\prime\}\)\\cdot base\+h\(\`b^\{\\prime\}\)\)\\cdot base,C​\(‘​‘​c​d′′\)\\displaystyle C\(\`\`cd^\{\\prime\\prime\}\)=\(h​\(‘​c′\)⋅b​a​s​e\+h​\(‘​d′\)\)⋅b​a​s​e\.\\displaystyle=\(h\(\`c^\{\\prime\}\)\\cdot base\+h\(\`d^\{\\prime\}\)\)\\cdot base\.Rearranging Eq\.[9](https://arxiv.org/html/2607.29065#Sx3.E9)shows that Eq\.[4](https://arxiv.org/html/2607.29065#Sx3.E4)≡\\equivEq\.[8](https://arxiv.org/html/2607.29065#Sx3.E8),

Hashp\(S\)=\[\(\(h\(‘a′\)⋅base\+h\(‘b′\)\)⋅base\)⋅base2\+\(\(h\(‘c′\)⋅base\+h\(‘d′\)\)⋅base\)⋅base0\]modM=\[C​\(‘​‘​a​b′′\)⋅B​\(\|‘​‘​c​d′′\|\)\+C​\(‘​‘​c​d′′\)\]modM=H​a​s​hp​\(T​\(S\)\)\.\\displaystyle\\begin\{split\}&Hash\_\{p\}\(S\)=\[\(\(h\(\`a^\{\\prime\}\)\\cdot base\+h\(\`b^\{\\prime\}\)\)\\cdot base\)\\cdot base^\{2\}\\\\ &\+\(\(h\(\`c^\{\\prime\}\)\\cdot base\+h\(\`d^\{\\prime\}\)\)\\cdot base\)\\cdot base^\{0\}\]\\mod M\\\\ &=\[C\(\`\`ab^\{\\prime\\prime\}\)\\cdot B\(\|\`\`cd^\{\\prime\\prime\}\|\)\+C\(\`\`cd^\{\\prime\\prime\}\)\]\\mod M\\\\ &=Hash\_\{p\}\(T\(S\)\)\.\\end\{split\}Simply substituting the hashing mechanism enables us to preserve similar algorithmic efficiency\. We use the general polynomial hashing approach for this work\. Other specific hashing approaches, such asBernstein \([2005](https://arxiv.org/html/2607.29065#bib.bib44)\); Degabrieleet al\.\([2024](https://arxiv.org/html/2607.29065#bib.bib47)\), may be applicable\.

### Research Questions

When comparing the different token sequences from different tokenizers, there are three key scenarios that will influence the training of tokenizer\-agnostic Engram embeddings\.

#### NN\-gram does not exist\.

ANN\-gram from a tokenizer A might never be found in any token sequence from tokenizer B\. This disparity can be largely attributed to different tokenization behaviours and vocabulary\. Theoretically, whenNNis large enough, there will be more chances to align byte\-equivalentNN\-grams from different tokenizers\. Additionally, most tokenizers preprocess texts similarly, e\.g\., splitting before space, and thus share similar points of partitions\. Nevertheless, are there enough byte\-equivalentNN\-grams between different tokenizers such that pretrained engram embeddings will be useful?

#### Joint embedding space for differentNN\-grams\.

Due toNN\-gram mismatch, different tokenizers might partition the same byte sequence into different numbers of tokens, e\.g\., a33\-gram from a target tokenizer might be a22\-gram of another reference tokenizer\. The original implementation modelsNN\-grams in a disjoint manner, i\.e\., embedding spaces between differentNNare disjoint\. This means that even if the exact byte\-equivalentNN\-gram had been learnt, due to the allocation of the embedding table, the information is inaccessible unlessNNis similar for both tokenizers\. A straightforward solution will involve removing the specificNN\-gram allocations and allow mixing between theNN\-grams of differentNN\. This would imply that the Engram embeddings will lose usefulNN\-related information, if any, and such information will be learnt upstream as the down\-projection of concatenatedNN\-gram embeddings remains unchanged\. Will thisNN\-gram mixing adversely affect the learning of Engram embeddings?

#### 11\-gram requirement\.

The original implementation does not model 1\-gram information as the backbone model is responsible for that\. Between different tokenizers, it is plausible that single tokens from one tokenizer will be partitioned into multiple tokens when using another tokenizer\. If we ignore 1\-grams in the Engram embedding space, there could be information loss when transferring the Engram embeddings to a model with a different tokenizer\. To include 1\-grams, there will be a small increase in parameters for the down\-projection to key/values\. Since 1\-gram hashes will be frequently accessed and updated during training, will they adversely affect the learning of Engram embeddings?

![Refer to caption](https://arxiv.org/html/2607.29065v1/x3.png)\(a\)% of uniqueNN\-grams fromMistraltokenizer found incl100k\_basetokenizer
![Refer to caption](https://arxiv.org/html/2607.29065v1/x4.png)\(b\)% of uniqueNN\-grams fromSmolLM2tokenizer found incl100k\_basetokenizer

Figure 2:Percentages of uniqueNN\-grams sampled on wikitextv​a​lvalusing different target tokenizers \(MistralandSmolLM2\) that exists when using reference tokenizer \(cl100k\_base\)\.NN\-gram pairs, on the diagonal, might be partitioned differently\.

## Experiments

### Backbone Language Models

For our experiments, we pretrain from scratch multiple variants using these three decoder\-only transformer backbones:333One run for each model variants on a 8xH200 GPU node, 4096 max length token packing, conv\. kernel size 4 for Engram module\.

1. 1\.SmolLM2\-1\.7B\(Allalet al\.[2025](https://arxiv.org/html/2607.29065#bib.bib19)\)\. The underlying architecture isLlama2\(Touvronet al\.[2023](https://arxiv.org/html/2607.29065#bib.bib23)\), with full SPDA attention blocks\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.29065#bib.bib15)\)\.
2. 2\.Qwen2\-7B\(Yanget al\.[2024](https://arxiv.org/html/2607.29065#bib.bib22)\)\. Uses Group Query Attention \(GQA\)\(Ainslieet al\.[2023](https://arxiv.org/html/2607.29065#bib.bib16)\)instead of full attention\.
3. 3\.Qwen3\.5\-0\.8B\-text\-only444principled\-intelligence/Qwen3\.5\-0\.8B\-text\-only\(Qwen Team[2026](https://arxiv.org/html/2607.29065#bib.bib20)\)\. While this model can process both image and text modalities, we only utilize its 0\.7B text\-relevant parameters\. It uses Gated Delta Net\(Yanget al\.[2025](https://arxiv.org/html/2607.29065#bib.bib21)\)blocks, at a ratio of 3:1 to transformer blocks\.

These models use mostly default settings from HuggingFace’s repository, with changes to use other tokenizers and to add Engram modules\.555Only two blocks: after first block and in middle of backbone\.We allocate∼\\sim35% of total parameters to the Engram module, similar toChenget al\.\([2026](https://arxiv.org/html/2607.29065#bib.bib31)\)\.666This parameter budget anchors our hyperparameters selection\.During pretraining, we use AdamW\(Loshchilov and Hutter[2019](https://arxiv.org/html/2607.29065#bib.bib49)\)optimizer with a cosine scheduler with warm up\.

### Benchmark and Metrics\.

We use several popular benchmarks, from EleutherAI’slm\-evaluation\-harness\(Gaoet al\.[2024](https://arxiv.org/html/2607.29065#bib.bib2)\), that are commonly\-used for pretraining evaluation: ARC\(Clarket al\.[2018](https://arxiv.org/html/2607.29065#bib.bib3)\), BoolQ\(Clarket al\.[2019](https://arxiv.org/html/2607.29065#bib.bib8)\), COPA\(Roemmeleet al\.[2011](https://arxiv.org/html/2607.29065#bib.bib13)\), HellaSwag\(Zellerset al\.[2019](https://arxiv.org/html/2607.29065#bib.bib4)\), LAMBADA\(Papernoet al\.[2016](https://arxiv.org/html/2607.29065#bib.bib9); Radfordet al\.[2019](https://arxiv.org/html/2607.29065#bib.bib17)\), PIQA\(Bisket al\.[2020](https://arxiv.org/html/2607.29065#bib.bib6)\), SCIQ\(Johannes Welbl[2017](https://arxiv.org/html/2607.29065#bib.bib7)\), and Winogrande\(Levesqueet al\.[2011](https://arxiv.org/html/2607.29065#bib.bib14); Sakaguchiet al\.[2019](https://arxiv.org/html/2607.29065#bib.bib12)\)\. When possible, for these multiple\-choice question benchmarks, we use length\-normalized accuracy \(acc\_norm\) instead of accuracy \(acc\)\.777Select choice candidate with the highest log\-prob sum\. We can normalize for tokens length to reduce bias towards long answers\.For aggregation across benchmarks, we report theMeanof all accuracy and accuracy norm values together\. Finally, as we will be comparing across tokenizers, we reportbits\-per\-byteofwikitextbenchmark\(Merityet al\.[2016](https://arxiv.org/html/2607.29065#bib.bib11)\)\.

### Byte\-equivalentNN\-grams Analysis

First, we examine the occurrences ofNN\-grams, from the target tokenizer, 1\) that do not exist in the reference tokenizer and 2\) byte\-equivalentNN\-grams with differentNN, i\.e\., found in both target and reference tokenizer\. We select three different tokenizers, withMistral\(∼\{\\sim\}32KKsize\)\(Jianget al\.[2023](https://arxiv.org/html/2607.29065#bib.bib18)\)andSmolLM2\(∼\{\\sim\}49KKsize\) tokenizers as target tokenizers, andcl100k\_base\(∼\{\\sim\}100KKsize\)\(OpenAI[2022](https://arxiv.org/html/2607.29065#bib.bib1)\)as the reference tokenizer\. We employ our target and reference tokenizers to tokenize wikitextv​a​lval, consisting of 641 documents totaling 2M tokens, obtaining uniqueNN\-grams varyingN∈\[1,7\]N\\in\[1,7\]\. We then compare the obtained uniqueNN\-gram sets, per document\. We expect some targetNN\-grams to be missing from the referenceNN\-grams\. A reference tokenizer with a larger tokenizer vocabulary tends to compress better, which may result in fewer uniqueNN\-grams\.

NNTarget1234567Mistral0\.800\.810\.800\.780\.750\.740\.70SmolLM20\.920\.910\.890\.860\.860\.850\.80

Table 2:% of unique wikitextv​a​lvalNN\-grams obtained from target tokenizer and can be found in the uniqueNN\-grams of reference tokenizer\. Refer to Fig\.[2](https://arxiv.org/html/2607.29065#Sx3.F2)for the breakdown\.#### Byte\-equivalentNN\-grams exists across tokenizers\.

From Table[2](https://arxiv.org/html/2607.29065#Sx4.T2), observe that there is a healthy percentage of uniqueNN\-grams, sampled from wikitextv​a​lval, from our target tokenizers and found in the reference tokenizer\. If these byte\-equivalentNN\-grams exist, this means that information stored in their respective Engram embeddings can be accessed despite the difference in tokenizers\.

#### Byte\-equivalentNN\-grams with differentNN\.

From our detailed breakdown in Figure[2](https://arxiv.org/html/2607.29065#Sx3.F2), observe that when targetNNincreases, the proportion of byte\-equivalentNN\-gram found in other referenceNNincreases substantially\. This implies that without enforcing the byte\-equivalence constraint acrossNN, we decrease the possible avenues for cross\-tokenizer information transfer viaNN\-grams\.

ExperimentEngram?CompareArccArceBoolQCOPAHella\.LAMB\.PIQASCIQWinoMean\(↑\)\(\\uparrow\)bits/byte\(↓\)\(\\downarrow\)Base 1\.7BNOReference0\.2770\.5340\.5810\.6700\.4720\.4820\.7040\.7580\.5230\.5560\.861Base 1\.7B\+ Engram 0\.8B\+ XORNO\-0\.3120\.5780\.6160\.7200\.5010\.4950\.7010\.7980\.5590\.5871\.034To Base \(%\)0\.1260\.0820\.0600\.0750\.0610\.027\-0\.0040\.0530\.0690\.0560\.201YES\-0\.3170\.6140\.5980\.7000\.5460\.5580\.7240\.8250\.5670\.6050\.919To Base \(%\)0\.1440\.1500\.0290\.0450\.1570\.1580\.0280\.0880\.0840\.0900\.067Base 1\.7B\+ Engram 0\.8B\+ PolyNO\-0\.3180\.6040\.5300\.7300\.5330\.5130\.7140\.8210\.5640\.5921\.025To Base \(%\)0\.1480\.131\-0\.0880\.0900\.1290\.0640\.0140\.0830\.0780\.0650\.190YES\-0\.3280\.6290\.5560\.7600\.5590\.5660\.7270\.8270\.5600\.6120\.925To Base \(%\)0\.1840\.178\-0\.0430\.1340\.1840\.1740\.0330\.0910\.0710\.1020\.074

Table 3:We pretrain from scratch a reference base model \(SmolLM2\-1\.7B\), and two Engram models differing in hashing mechanism: original XOR hashing and the proposed Polynomial hashing substitution\. We train the models using 32BSmolLM2tokens fromdclm\-dedupcorpus\. The models using XOR and Polynomial have similar training hyperparameters\. Results here shows that using Polynomial hashing does not negatively affect training and have comparable results to XOR\-based hashing\. For the models with Engram, we additionally evaluate its backbone without the Engram module to ascertain its value\-add\. Engram? NO indicates that we switch off the Engram parameters\. Standard error for all accuracy\-metrics benchmarks≤0\.015\\leq 0\.015\.

### Comparing XOR and Poly\. Hashing Engrams

For this experiment in this subsection, we wish to determine whether the requirements of 1\)11\-gram and 2\) joint embedding spaces will affect the pretraining of Engram modules\. We train three models usingSmolLM2\-1\.7Bas the backbone with its native tokenizer, on 32B tokens fromdclm\-dedup\(Tokpanovet al\.[2024](https://arxiv.org/html/2607.29065#bib.bib24)\):

1. a\.Base\-1\.7B\. As the main comparisons of our subsequent models are along various dimensions, this model serves as a*common*point of reference and normalizes the results for easier comparison\. The difference between the subsequent models and the base models highlights the value add from the additional Engram module, giving a better perspective compared to a direct comparison\.
2. b\.Additional 0\.8B Engram module with hyperparameterN=3N=3, 1M indices of size 192 per block\. 1. i\.Using the original XOR hashing, we adapt DeepSeek’s demonstration code888Original demo code: github\.com/deepseek\-ai/Engramfor our training infrastructure\. 2. ii\.Using our proposed polynomial hashing, there are three main changes\. First, we directly substitute the hashing algorithm\. Second, we remove the disjoint embedding spaces by using the sameb​a​s​ebasepositional hyperparameter for allNN\. Lastly, we includeN=1N=1hash indices per token\. To reduce the impact ofN=1N=1at the start of training, we scale its embedding values by0\.10\.1\. Note that as training progress, ifN=1N=1is indeed informative, this scaling will be made redundant\.

ExperimentEngram?CompareArccArceBoolQCOPAHella\.LAMB\.PIQASCIQWinoMean\(↑\)\(\\uparrow\)bits/byte\(↓\)\(\\downarrow\)ModelA7B\+ Engram 2BNO\-0\.3300\.6170\.5400\.7800\.5840\.5510\.7280\.8370\.5790\.6160\.779YES\-0\.3460\.6600\.5940\.7700\.6270\.6110\.7570\.8580\.5900\.6460\.708ModelB0\.7BNOReference0\.2720\.5390\.5530\.7000\.4580\.4990\.6900\.7720\.5370\.5600\.821ModelB0\.7B\+ Pretrained ModelA’sEngram Emb\. 1\.5BNO\-0\.2560\.5090\.5710\.7000\.4370\.3560\.6860\.7100\.5250\.5280\.905To B \(%\)\-0\.059\-0\.0560\.0330\.000\-0\.046\-0\.287\-0\.006\-0\.080\-0\.022\-0\.0630\.102YES\-0\.270\.5450\.6110\.7500\.4760\.5140\.7030\.8000\.5370\.5840\.811To B \(%\)\-0\.0070\.0110\.1050\.0710\.0390\.0300\.0190\.0360\.0000\.038\-0\.012

Table 4:We pretrain from scratch aQwen2\-7B\(ModelA\), with an 2B Engram module \(N=3N=3\), on 150B tokens fromdclm\-dedup, tokenized usingcl100k\_basetokenizer\. After training ModelA\(Qwen3\.5\-0\.8B\-text\-only\), we use the pretrained Engram embeddings as frozen parameters, connected via a \(NN= 7\) module, to pretrain ModelB\. ModelBwas trained on 100BSmolLM2tokens fromdclm\-dedup\. Variants of ModelBuses similar hyperparameters\. Results shows that ModelBcan use ModelA’s pretrained Engram embeddings\. Standard error for all accuracy\-metrics benchmarks≤0\.015\\leq 0\.015\.We compare the inter\-hashing results between the different hashing approaches and intra\-hashing results where we isolate the contribution of the Engram module\. If the additional Engram module is redundant, we expect no performance difference when the Engram module is deactivated, as the model is wholly reliant on the backbone model\.

#### Results\.

As reported in Table[3](https://arxiv.org/html/2607.29065#Sx4.T3), for our first comparison, we examine the difference between models with different hashing approaches\. With the exception of BoolQ999Latter experiments suggests this is resolved via scaling tokens\., the results of polynomial hashing, from other benchmarks, are comparable to that of XOR hashing\. This suggests that modifications for11\-gram and joint\-embedding spaces did not degrade pretraining performance\. When we compare the results between active and inactive engram modules, within the same Engram models, we observe large improvements in mean and bits\-per\-byte which suggests that there are contributions from the additional Engram modules\.

#### Discussion\.

From this empirical experiment, we can conclude that Engram modules are useful in both Engram models\. The inclusion of11\-gram does not seem to negatively impact training, which we will again verify in our subsequent ablation\. The key benefit of modelling disjoint embedding spaces allows us to designate the proportion of embedding parameters to specificNN, controlling the importance of specificNN\. A larger proportion assigned will decrease the chances of hash collision for that specificNN\-gram\. Our experiment assumed equal proportion\. The context\-aware SPDA mechanism, in the Engram module, serves to gate these collisions\. Hence, collisions acrossNNmay not matter as much or are mitigated by using the complete embedding table\. The similarity of the results suggests that the useful information lies at the byte\-level and by happenstance in theNN\-gram space, and the main role ofNN\-grams is to sample and shortlist potential useful byte sequences\.

ExperimentEngram?ArccArceBoolQCOPAHella\.LAMB\.PIQASCIQWinoMean\(↑\)\(\\uparrow\)bits/byte\(↓\)\(\\downarrow\)ModelC\(N=1\)NO0\.2530\.4930\.6150\.6600\.4320\.4550\.6870\.7420\.5180\.5420\.851YES0\.2540\.4910\.6150\.6700\.4310\.4570\.6910\.7430\.5140\.5440\.851Difference YES/NO \(%\)0\.004\-0\.0040\.0000\.015\-0\.0020\.0040\.0060\.001\-0\.0080\.0030\.000ModelB\(N=7\)NO0\.2440\.4950\.5960\.7100\.4260\.4020\.6870\.7000\.5150\.5330\.872YES0\.2520\.5010\.6150\.6800\.4380\.4550\.6850\.7390\.5300\.5460\.845Difference YES/NO \(%\)0\.0330\.0120\.032\-0\.0420\.0280\.132\-0\.0030\.0560\.0290\.025\-0\.031

Table 5:Ablating results from Table[4](https://arxiv.org/html/2607.29065#Sx4.T4), we have Engram ModelC, of similar architecture and tokenizer to Engram ModelB, with the exception of settingN=1N=1\. ModelCis pretrained on 32B tokens and compared to an equivalent checkpoint of ModelB\. Results from this table show that there is little contribution from the Engram module whenN=1N=1, implying that the improvements came from usingNN\-grams ofN\>1N\>1\. Standard error for all accuracy\-metrics benchmarks≤0\.015\\leq 0\.015\.

### Training Tokenizer\-agnostic Engrams

Our reference tokenizer iscl100k\_base, and our target tokenizer isSmolLM2\. This experiment examines whether our proposed hashing approach allows a model with a target tokenizer to use Engram embeddings trained from a different reference tokenizer\. We pretrain ModelA: aQwen2\-7Bmodel with a 2B Engram module \(N=3N=3, 2M indices of size 192 per block\) on 150B reference tokens fromdclm\-dedup\. Then, we pretrain two variants of a smaller ModelBbased onQwen3\.5\-0\.8B\-text\-onlybackbone:

1. a\.No Engram module, serving as the point of reference\.
2. b\.We attach an Engram module \(N=7N=7\)101010We select a highNNto capture as manyNN\-grams from the reference tokenizer\. It is not necessarily advantageous to use a highNNdue to hash collisions in the embedding space\., that uses the previously pretrained engram embeddings as frozen parameters\. Note that the other∼\\sim15M parameters in the Engram module are trainable\.

Both variants of ModelBare trained on 100B target tokens fromdclm\-dedupin the same order flow\. Our previous analysis show that there are many byte\-equivalentNN\-grams across tokenizers\. If these commonNN\-grams are meaningful lookups to the pretrained Engram embeddings, we expect to observe an improvement in our selected benchmarks, indicating the value add of the additional parameters\.

#### Results\.

Between the Engram module activations and the reference model, in Table[4](https://arxiv.org/html/2607.29065#Sx4.T4), we observe that ModelB\+Engram shows improvements in most benchmarks, notably in BoolQ \(\+10%\) and COPA \(\+7%\)\. Without activating the Engram module, benchmark performance decreases, suggesting that the commonNN\-grams are meaningful and cross\-tokenizer lookups on pretrained Engram embeddings are successful, evidence of tokenizer agnosticism\.

### Ablation Study

Since our proposed method includes11\-grams, could the model have relied on11\-grams to achieve better performance? We introduce another Engram ModelC, similar to Engram ModelB, except that we setN=1N=1, essentially allowing the model to access only byte\-equivalent11\-grams found in theSmolLM2tokenizer\. We pretrain Engram ModelCon 32B, on the same data flow as the corresponding checkpoint of the Engram ModelB\.111111We intially pretrain ModelA,B, andCto 32B tokens, before deciding to scale ModelAandBto 100B tokens\.

![Refer to caption](https://arxiv.org/html/2607.29065v1/x5.png)Figure 3:Plots of difference in training loss between Engram Models with pretrained Engram embeddings and reference model, against the 100B token training run\. Each plot indicates an additional 2B tokens\.#### Results\.

With the results in Table[5](https://arxiv.org/html/2607.29065#Sx4.T5), we can see that for ModelC, there is only a marginal difference between activating and deactivating the engram module, compared to the more substantial differences from ModelB\. This indifference suggests that for ModelC, its Engram module was not beneficial\. We further examine the training loss curves in Figure[3](https://arxiv.org/html/2607.29065#Sx4.F3), which shows that the additional engram parameters, for Model𝐂\\mathbf\{C\}, were counterproductive\. This rules out that the improvements were from11\-grams, and that\(N\>1\)\(N\>1\)\-grams had successful lookups to the pretrained Engram embeddings\. Comparing between Engram module activation modes, this evaluation methodology shows that having more parameters does not necessarily lead to better performance\.

## Conclusion and Future Work

Reusing, modifying, and improving pretrained weights constitute an important and prevalent direction for cost\-effective academic research\. We highlight the substitutability of byte\-level and token\-levelNN\-gram information, to remove the tight tokenizer coupling from Engram module\. To achieve this, we propose using the general polynomial hashing approach to attain hash equivalence for byte\-equivalent token sequences\. This requires us to modify the Engram module to accept 1\-gram information and to modelNN\-grams in a shared embedding space\. Our experiments show that these modifications are reasonable and enables effective cross\-tokenizer transfer\. Possible future work directions involve scaling Engram in a tokenizer\-agnostic manner\. Beyond the dominant English language, it will also be interesting to investigate the multilingual capability of its hashed embeddings\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.arXiv preprint arXiv:2305\.13245\.Cited by:[item 2](https://arxiv.org/html/2607.29065#Sx4.I5.i2.p1.1)\.
- L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav, J\. Lochner, C\. Fahlgren, X\. Nguyen, C\. Fourrier, B\. Burtenshaw, H\. Larcher, H\. Zhao, C\. Zakka, M\. Morlon, C\. Raffel, L\. von Werra, and T\. Wolf \(2025\)SmolLM2: when smol goes big – data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[item 1](https://arxiv.org/html/2607.29065#Sx4.I5.i1.p1.1)\.
- D\. J\. Bernstein \(2005\)The poly1305\-AES message\-authentication code\.InFast Software Encryption: 12th International Workshop, FSE 2005, Paris, France, February 21–23, 2005, Revised Selected Papers,H\. Gilbert and H\. Handschuh \(Eds\.\),Lecture Notes in Computer Science, Vol\.3557,pp\. 32–49\.External Links:ISBN 3\-540\-26541\-4Cited by:[Example\.](https://arxiv.org/html/2607.29065#Sx3.SSx3.SSS0.Px1.p1.5)\.
- S\. Bhattacharyya, K\. Nath, and P\. Sarkar \(2025\)Polynomial hashing over prime order fields\.Advances in Mathematics of Communications19\(1\),pp\. 337–378\.External Links:ISSN 1930\-5346,[Document](https://dx.doi.org/10.3934/amc.2024001),[Link](https://www.aimsciences.org/article/id/65a63bba92d3ad47ddcdd5f0)Cited by:[Polynomial\-hashing as an Efficient Alternative](https://arxiv.org/html/2607.29065#Sx3.SSx3.p1.4)\.
- Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.InThirty\-Fourth AAAI Conference on Artificial Intelligence,Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- T\. Brants, A\. C\. Popat, P\. Xu, F\. J\. Och, and J\. Dean \(2007\)Large language models in machine translation\.InProceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning \(EMNLP\-CoNLL\),J\. Eisner \(Ed\.\),Prague, Czech Republic,pp\. 858–867\.External Links:[Link](https://aclanthology.org/D07-1090/)Cited by:[NN\-gram modelling an illusion?](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px3.p1.11)\.
- C\. Buck, K\. Heafield, and B\. van Ooyen \(2014\)N\-gram counts and language models from the Common Crawl\.InProceedings of the Ninth International Conference on Language Resources and Evaluation \(LREC’14\),N\. Calzolari, K\. Choukri, T\. Declerck, H\. Loftsson, B\. Maegaard, J\. Mariani, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Reykjavik, Iceland,pp\. 3579–3584\.External Links:[Link](https://aclanthology.org/L14-1074/)Cited by:[NN\-gram modelling an illusion?](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px3.p1.11)\.
- J\. Carter and M\. N\. Wegman \(1979\)Universal classes of hash functions\.Journal of Computer and System Sciences18\(2\),pp\. 143–154\.External Links:ISSN 0022\-0000,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0022-0000%2879%2990044-8),[Link](https://www.sciencedirect.com/science/article/pii/0022000079900448)Cited by:[Polynomial\-hashing as an Efficient Alternative](https://arxiv.org/html/2607.29065#Sx3.SSx3.p1.4)\.
- X\. Cheng, R\. Tian, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie, K\. Huang, X\. Yu, C\. Deng, S\. Zhou, C\. Zhao, Z\. Hao, Y\. Li, H\. Zhang, Z\. Zhang, Y\. Wei, M\. Y\. Xu, H\. Zhang, D\. Zhao, and W\. Liang \(2026\)Conditional memory via scalable lookup: a new axis of sparsity for large language models\.Cited by:[DeepSeek’s Engram Module\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px1.p1.5),[1\(a\)](https://arxiv.org/html/2607.29065#Sx2.F1.sf1),[Hash Embeddings\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px3.p1.1),[Backbone Language Models](https://arxiv.org/html/2607.29065#Sx4.SSx1.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InNAACL,Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- J\. P\. Degabriele, J\. Gilcher, J\. Govinden, and K\. G\. Paterson \(2024\)SoK: efficient design and implementation of polynomial hash functions over prime fields\.In2024 IEEE Symposium on Security and Privacy \(SP\),Vol\.,pp\. 3128–3146\.External Links:[Document](https://dx.doi.org/10.1109/SP54263.2024.00132)Cited by:[Example\.](https://arxiv.org/html/2607.29065#Sx3.SSx3.SSS0.Px1.p1.5)\.
- P\. Gage \(1994\)A new algorithm for data compression\.C Users J\.12\(2\),pp\. 23–38\.External Links:ISSN 0898\-9788,[Link](https://dl.acm.org/doi/10.5555/177910.177914)Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- Google Team \(2025\)Gemma 3n\.External Links:[Link](https://ai.google.dev/gemma/docs/gemma-3n)Cited by:[Engram Module Adjacent\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.\(2023\)Mistral 7B\.arXiv preprint arXiv:2310\.06825\.Cited by:[Byte\-equivalentNN\-grams Analysis](https://arxiv.org/html/2607.29065#Sx4.SSx3.p1.17)\.
- M\. G\. Johannes Welbl \(2017\)Crowdsourcing multiple choice science questions\.arXiv:1707\.06209v1\.Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- T\. Kudo and J\. Richardson \(2018\)SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,E\. Blanco and W\. Lu \(Eds\.\),Brussels, Belgium,pp\. 66–71\.External Links:[Link](https://aclanthology.org/D18-2012),[Document](https://dx.doi.org/10.18653/v1/D18-2012)Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- T\. Kudo \(2018\)Subword regularization: improving neural network translation models with multiple subword candidates\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 66–75\.Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- H\. J\. Levesque, E\. Davis, and L\. Morgenstern \(2011\)The Winograd schema challenge\.InAAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning,Vol\.46,pp\. 47\.Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- A\. Liu, J\. Hayase, V\. Hofmann, S\. Oh, N\. A\. Smith, and Y\. Choi \(2025\)SuperBPE: space travel for language models\.InTokenization Workshop,External Links:[Link](https://openreview.net/forum?id=LwTWkSXIpt)Cited by:[Tokenization\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px1.p1.3)\.
- J\. Liu, S\. Min, L\. Zettlemoyer, Y\. Choi, and H\. Hajishirzi \(2024\)Infini\-gram: scaling unbounded n\-gram language models to a trillion tokens\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=u2vAyMeLMm)Cited by:[NN\-gram modelling an illusion?](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px3.p1.11)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[Backbone Language Models](https://arxiv.org/html/2607.29065#Sx4.SSx1.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv:1609\.07843\.Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- W\. Merrill, N\. A\. Smith, and Y\. Elazar \(2024\)Evaluatingnn\-gram novelty of language models using rusty\-DAWG\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 14459–14473\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.800/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.800)Cited by:[NN\-gram modelling an illusion?](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px3.p1.11)\.
- T\. Nguyen \(2024\)Understanding transformers via n\-gram statistics\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=WCc440cUhX)Cited by:[NN\-gram modelling an illusion?](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px3.p1.11)\.
- OpenAI \(2022\)Tiktoken: fast bpe tokenizer for use with openai’s models\.External Links:[Link](https://github.com/openai/tiktoken)Cited by:[Byte\-equivalentNN\-grams Analysis](https://arxiv.org/html/2607.29065#Sx4.SSx3.p1.17)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[item 3](https://arxiv.org/html/2607.29065#Sx4.I5.i3.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.External Links:[Link](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- M\. Roemmele, C\. A\. Bejan, and A\. S\. Gordon \(2011\)Choice of plausible alternatives: an evaluation of commonsense causal reasoning\.In2011 AAAI Spring Symposium Series,Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- R\. Sadhukhan, S\. Cao, H\. Dong, C\. Zhao, A\. Purpura\-Pontoniere, Y\. Tian, Z\. Liu, and B\. Chen \(2026\)STEM: SCALING TRANSFORMERS WITH EMBEDDING MODULES\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gufRimweSQ)Cited by:[Engram Module Adjacent\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px2.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2019\)WinoGrande: an adversarial winograd schema challenge at scale\.arXiv preprint arXiv:1907\.10641\.Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- C\. W\. Schmidt, V\. Reddy, C\. Tanner, and Y\. Pinter \(2025\)Boundless byte pair encoding: breaking the pre\-tokenization barrier\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=oPAjXGV8qQ)Cited by:[Tokenization\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px1.p1.3)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- D\. Svenstrup, J\. M\. Hansen, and O\. Winther \(2017\)Hash embeddings for efficient word representations\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 4935–4943\.External Links:ISBN 9781510860964Cited by:[Hash Embeddings\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px3.p1.1)\.
- S\. Takase, R\. Ri, S\. Kiyono, and T\. Kato \(2025\)Large vocabulary size improves large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 1015–1026\.External Links:[Link](https://aclanthology.org/2025.findings-acl.57/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.57),ISBN 979\-8\-89176\-256\-5Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- C\. Tao, Q\. Liu, L\. Dou, N\. Muennighoff, Z\. Wan, P\. Luo, M\. Lin, and N\. Wong \(2024\)Scaling laws with vocabulary: larger models deserve larger vocabularies\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=sKCKPr8cRL)Cited by:[Tokenizer lock in\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px2.p1.1)\.
- Y\. Tokpanov, P\. Glorioso, A\. Dattagupta, V\. Jawa, R\. Wolf, V\. Jeyakumar, A\. Mehta, Q\. Anthony, and B\. Millidge \(2024\)Building Zyda\-2, a 5 Trillion Token High\-Quality Dataset, with NVIDIA NeMo Curator\.External Links:[Link](https://www.zyphra.com/post/building-zyda-2)Cited by:[Comparing XOR and Poly\. Hashing Engrams](https://arxiv.org/html/2607.29065#Sx4.SSx4.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.Cited by:[item 1](https://arxiv.org/html/2607.29065#Sx4.I5.i1.p1.1)\.
- A\. Tseng and C\. D\. Sa \(2026\)$L^3$: large lookup layers\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=A7LiBHEsOo)Cited by:[Engram Module Adjacent\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px2.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.arXiv preprint arXiv:1706\.03762\.Cited by:[DeepSeek’s Engram Module\.](https://arxiv.org/html/2607.29065#Sx1.SS0.SSS0.Px1.p1.5),[item 1](https://arxiv.org/html/2607.29065#Sx4.I5.i1.p1.1)\.
- A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Yang, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, X\. Liu, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Guo, and Z\. Fan \(2024\)Qwen2 technical report\.Cited by:[item 2](https://arxiv.org/html/2607.29065#Sx4.I5.i2.p1.1)\.
- S\. Yang, J\. Kautz, and A\. Hatamizadeh \(2025\)Gated delta networks: improving mamba2 with delta rule\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=r8H7xhYPwz)Cited by:[item 3](https://arxiv.org/html/2607.29065#Sx4.I5.i3.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Cited by:[Benchmark and Metrics\.](https://arxiv.org/html/2607.29065#Sx4.SSx2.p1.1)\.
- Y\. Zheng, G\. Xia, X\. Wang, and L\. Ren \(2026\)Lngram: n\-gram conditional memory in latent space\.arXiv preprint arXiv:2605\.24869\.Cited by:[Engram Module Adjacent\.](https://arxiv.org/html/2607.29065#Sx2.SS0.SSS0.Px2.p1.1)\.