MultiHashFormer: Hash-based Generative Language Models
Summary
MultiHashFormer is a hash-based generative language model that represents each token as a unique hash signature, enabling parameter-efficient autoregression. It outperforms standard Transformer LMs at 100M, 1B, and 3B scales and supports multilingual vocabulary expansion without increasing parameters.
View Cached Full Text
Cached at: 06/29/26, 05:25 AM
# MultiHashFormer: Hash-based Generative Language Models Source: [https://arxiv.org/abs/2606.28057](https://arxiv.org/abs/2606.28057) [View PDF](https://arxiv.org/pdf/2606.28057) > Abstract:Language models \(LMs\) represent tokens using embedding matrices that scale linearly with the vocabulary size\. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder\-only models\. While this offers parameter efficiency, many\-to\-one collisions prevent its use in causal LMs\. In this paper, we propose MultiHashFormer, a new framework that allows hash\-based autoregression\. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions\. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder\. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text\. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks\. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications\. ## Submission history From: Huiyin Xue \[[view email](https://arxiv.org/show-email/3c52616c/2606.28057)\] **\[v1\]**Fri, 26 Jun 2026 13:03:29 UTC \(4,031 KB\)
Similar Articles
Multi-Head Attention Residuals
Introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into per-subspace heads so each feature subspace reads depth history via its own softmax. Trained from scratch on a Nemotron-based corpus, MHAR consistently improves validation loss over standard Transformers from 100M to 1B scale and also boosts mid-training downstream accuracy.
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
This paper introduces BitLM, a language model that uses bitwise continuous diffusion to generate multiple tokens in parallel, aiming to overcome the sequential bottleneck of traditional autoregressive generation while preserving causal structure.
PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics
This paper introduces PermuFormer, an autoregressive transformer pretrained on multi-task, multi-encoding data for permutation-focused tasks in algebraic combinatorics, demonstrating effective fine-tuning on downstream tasks compared to baselines.
Dynamic Multi-Byte Prediction With Hierarchical Language Models
The paper introduces multi-byte prediction to speed up inference in byte-level language models by generating multiple bytes in parallel with minimal performance impact.
Multi-Block Diffusion Language Models
This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.