BitNet Text Embeddings

# BitNet Text Embeddings
Source: [https://arxiv.org/html/2606.25674](https://arxiv.org/html/2606.25674)
Zhen Li1Xin Huang2Liang Wang2Nan Yang2Ting Song2Yan Xia2Xun Wu2& Shaohan Huang2Huishuai Zhang1Furu Wei2Dongyan Zhao1 1Peking University2Microsoft Research [https://aka\.ms/GeneralAI](https://aka.ms/GeneralAI)

###### Abstract

LLM\-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high\-dimensional full\-precision embeddings impose substantial storage and bandwidth overhead on large\-scale indexes\. In this paper, we presentBitEmbed, an extreme low\-bit framework for LLM\-based text embedding that jointly targets encoding efficiency and vector storage\.BitEmbedconverts pretrained LLM backbones into BitNet\-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement\. The converted model is adapted to representation learning through continual contrastive pre\-training, followed by supervised contrastive fine\-tuning with both similarity\-distribution distillation and attention\-relation distillation from a full\-precision teacher\. Beyond quantizing the backbone,BitEmbedfurther trains output embeddings to support multiple storage precisions meeting different storage needs in various scenarios\. Experiments on MMTEB \(eng, v2\) with Qwen3\-0\.6B and Gemma3\-270M show thatBitEmbedis largely comparable to full precision teacher embedders\. Moreover,BitEmbedflexibly obtains text embeddings of various precisions, achieving a trade\-off between performance and storage cost\.

## 1Introduction

Text embeddings have become a core interface between natural language and large\-scale information systems\. Modern retrieval\[[51](https://arxiv.org/html/2606.25674#bib.bib108)\], retrieval\-augmented generation\[[77](https://arxiv.org/html/2606.25674#bib.bib111)\], recommendation\[[79](https://arxiv.org/html/2606.25674#bib.bib192)\], and question answering\[[7](https://arxiv.org/html/2606.25674#bib.bib107)\]pipelines all rely on embedding models to map text into dense vectors whose geometry reflects semantic relevance\[[54](https://arxiv.org/html/2606.25674#bib.bib193),[25](https://arxiv.org/html/2606.25674#bib.bib129),[20](https://arxiv.org/html/2606.25674#bib.bib157),[49](https://arxiv.org/html/2606.25674#bib.bib140)\]\. Pre\-trained bidirectional encoder architectures such as BERT\[[15](https://arxiv.org/html/2606.25674#bib.bib136)\]have been widely adopted as backbone models for text embedding\[[24](https://arxiv.org/html/2606.25674#bib.bib109),[71](https://arxiv.org/html/2606.25674#bib.bib113),[78](https://arxiv.org/html/2606.25674#bib.bib112)\]\. With recent advances in large language models, LLM\-based embedders utilize foundation LLMs as the backbone and adapt them with large\-scale contrastive learning objectives\[[33](https://arxiv.org/html/2606.25674#bib.bib180),[57](https://arxiv.org/html/2606.25674#bib.bib134),[41](https://arxiv.org/html/2606.25674#bib.bib114),[29](https://arxiv.org/html/2606.25674#bib.bib142),[46](https://arxiv.org/html/2606.25674#bib.bib176)\]\. With the richer world knowledge and text understanding abilities inherent in LLMs, LLM\-based embedders have achieved strong performance across semantic representation tasks\. However, despite these advances, LLM\-based embedders still inherit the critical deployment challenges: high embedding inference latency and substantial storage costs\.

Encoding queries and documents with LLM backbones is considerably more expensive than with conventional encoder\-only models\. Meanwhile, the produced embeddings must be stored, transferred, and searched at massive scale\. A production retrieval system may maintain vectors for millions or billions of documents where high\-dimensional and high\-precision embeddings can dominate memory footprint and data movement even when the encoder itself is optimized\. These costs become more pronounced as LLM\-based embedders are adopted in online retrieval services and storage\-constrained scenarios, making efficient inference and representation central requirements for practical deployment\.

Extreme low\-bit LLMs provide a promising path toward reducing inference cost\[[14](https://arxiv.org/html/2606.25674#bib.bib194),[34](https://arxiv.org/html/2606.25674#bib.bib202),[23](https://arxiv.org/html/2606.25674#bib.bib200),[21](https://arxiv.org/html/2606.25674#bib.bib201),[5](https://arxiv.org/html/2606.25674#bib.bib195),[68](https://arxiv.org/html/2606.25674#bib.bib196)\]\. BitNet\-style models employ low\-precision ternary weights and quantized activations, enabling substantial memory savings and faster inference on suitable hardware while maintaining high precision for the optimizer states and gradients during training\[[62](https://arxiv.org/html/2606.25674#bib.bib199),[40](https://arxiv.org/html/2606.25674#bib.bib197),[39](https://arxiv.org/html/2606.25674#bib.bib198)\]\. BitNet Distillation further shows that full\-precision LLMs can be transformed into 1\.58\-bit task models through architectural stabilization, continued training, and distillation\[[69](https://arxiv.org/html/2606.25674#bib.bib203)\]\. Inspired by these findings, we explore whether similar low\-bit principles can be specialized for text embedding\.

In this paper, we introduceBitNet Text Embeddings, abbreviated asBitEmbed, a framework for extreme low\-bit LLM\-based text embedding\.BitEmbedconverts pre\-trained LLM backbones into BitNet\-style embedders, together with normalization modules that stabilize training under quantization\. To adapt the quantized backbone to representation learning,BitEmbedfirst performs continual contrastive pre\-training on large\-scale text pairs and then conducts supervised fine\-tuning with in\-batch and hard negatives\. Meanwhile, a full\-precision fp16 embedding model is tuned as a teacher model, where the quantized embedder is trained not only with contrastive supervision but also to match the teacher’s batchwise cosine\-similarity distribution and attention relations through distillation\. Beyond quantizing the backbone,BitEmbedalso trains the output embeddings to support multiple storage precisions\. Inspired by Matryoshka\-style representation and quantization\[[26](https://arxiv.org/html/2606.25674#bib.bib205),[58](https://arxiv.org/html/2606.25674#bib.bib139),[50](https://arxiv.org/html/2606.25674#bib.bib204)\], we quantize each embedding dimension to multiple precisions during training and optimize the average loss across these precisions\. This design encourages a single embedder to produce vectors that remain effective under different memory budgets, allowing practitioners to trade storage for quality without retraining separate models\.

We evaluateBitEmbedon the Massive Multilingual Text Embedding Benchmark \(MMTEB\)\[[17](https://arxiv.org/html/2606.25674#bib.bib130)\]\. OurBitEmbedapproaches the performance of full\-precision teachers while improving inference efficiency\. Furthermore,BitEmbedalso enables the trade\-off between storage cost and performance with multi\-precision embedding quantization\.

Specifically, our contributions are summarized as follows:

1. 1\.We present, to the best of our knowledge, the first systematic study of extreme low\-bit quantization for LLM\-based text embedding, targeting both encoder inference and vector\-index storage\.
2. 2\.We propose an effective BitNet\-style training framework that combines contrastive continual pre\-training, supervised fine\-tuning, similarity distribution and attention relation distillation\.
3. 3\.We introduce multi\-precision embedding training and verify thatBitEmbedcan approach full\-precision teacher performance while supporting efficient multi\-precision embedding representations\.

## 2Related Work

### 2\.1LLM\-based Text Embedders

Text embedders are increasingly moving from encoder\-only Transformer models\[[54](https://arxiv.org/html/2606.25674#bib.bib193),[25](https://arxiv.org/html/2606.25674#bib.bib129),[20](https://arxiv.org/html/2606.25674#bib.bib157),[63](https://arxiv.org/html/2606.25674#bib.bib206),[35](https://arxiv.org/html/2606.25674#bib.bib132)\]toward decoder\-only LLM backbones\[[49](https://arxiv.org/html/2606.25674#bib.bib140),[33](https://arxiv.org/html/2606.25674#bib.bib180),[64](https://arxiv.org/html/2606.25674#bib.bib117),[30](https://arxiv.org/html/2606.25674#bib.bib116),[58](https://arxiv.org/html/2606.25674#bib.bib139)\]\. Recently, LLM\-based text embedders have further explored instruction\-following, multilinguality, multi\-task generalization capabilities\[[29](https://arxiv.org/html/2606.25674#bib.bib142),[28](https://arxiv.org/html/2606.25674#bib.bib178),[8](https://arxiv.org/html/2606.25674#bib.bib211),[22](https://arxiv.org/html/2606.25674#bib.bib187)\]\. INSTRUCTOR\[[57](https://arxiv.org/html/2606.25674#bib.bib134)\]trains embeddings to follow natural\-language task instructions, enabling task\-specific representations through prompts\. Recent models also rely on more sophisticated training recipes\[[11](https://arxiv.org/html/2606.25674#bib.bib181),[56](https://arxiv.org/html/2606.25674#bib.bib208),[47](https://arxiv.org/html/2606.25674#bib.bib209),[3](https://arxiv.org/html/2606.25674#bib.bib210),[80](https://arxiv.org/html/2606.25674#bib.bib212)\]\. Qwen3\-Embedding\[[76](https://arxiv.org/html/2606.25674#bib.bib147)\]uses LLMs to synthesize large\-scale high\-quality data for multi\-stage contrastive training\. These studies focus primarily on improving embedding quality and generalization\. In contrast,BitEmbedstudies how to make LLM\-based embedders practical under strict inference and storage budgets by combining extreme low\-bit quantization with multi\-precision output embeddings\.

### 2\.2Quantization for LLMs

Quantization techniques are crucial for deploying Large Language Models \(LLMs\) under computational and storage constraints\. A common strategy is Post\-Training Quantization \(PTQ\)\[[14](https://arxiv.org/html/2606.25674#bib.bib194),[75](https://arxiv.org/html/2606.25674#bib.bib220),[70](https://arxiv.org/html/2606.25674#bib.bib213)\], encompassing methods like GPTQ\[[19](https://arxiv.org/html/2606.25674#bib.bib214)\]and AWQ\[[36](https://arxiv.org/html/2606.25674#bib.bib215)\]\. PTQ is highly efficient as it applies low\-bit quantization using only a small set of calibration data\. While PTQ methods offer rapid compression, they typically suffer from significant performance degradation at ultra\-low bitwidths below 4 bits\. To overcome this, Quantization\-Aware Training \(QAT\)\[[16](https://arxiv.org/html/2606.25674#bib.bib217),[38](https://arxiv.org/html/2606.25674#bib.bib218),[72](https://arxiv.org/html/2606.25674#bib.bib219),[9](https://arxiv.org/html/2606.25674#bib.bib216)\]integrates quantization directly into the optimization loop, significantly raising the performance ceiling for low\-bit models\. Recently, QAT has been pushed to extreme limits with 1\.58\-bit \(ternary\) architectures like BitNet\[[62](https://arxiv.org/html/2606.25674#bib.bib199),[40](https://arxiv.org/html/2606.25674#bib.bib197),[39](https://arxiv.org/html/2606.25674#bib.bib198)\]\. Inspired by Bitnet distllation\[[69](https://arxiv.org/html/2606.25674#bib.bib203)\], we introduce the 1\.58\-bit LLM\-based text embedder, bridging ternary QAT and representation distillation\.

## 3Method

BitEmbedis an extreme low\-bit LLM\-based text embedder which follows a three\-stage training pipeline\. Firstly, we convert a pretrained LLM into a BitNet\-style embedding encoder\. Then we adapt the converted encoder with continual contrastive pre\-training on large\-scale text pairs\. Finally we fine\-tune the low\-bit encoder on supervised training data using contrastive learning with teacher\-guided distillation from a FP16 embedding teacher\. Additionally, we expose output embeddings to multiple storage precisions, enabling a singleBitEmbedcheckpoint to support different memory budgets\.

### 3\.1LLM\-based Text Embedding Formulation

Given the input sequenceX=\{x1,x2,…,xn\}X=\\\{x\_\{1\},x\_\{2\},\.\.\.,x\_\{n\}\\\}wherexix\_\{i\}represents theii\-th token in the sequence, the decoder\-only LLMMMprocesses this sequence to produce a contextualized representation for each token, denoted asH​\(X\)=\{h1,h2,…,hn\}H\(X\)=\\\{h\_\{1\},h\_\{2\},\.\.\.,h\_\{n\}\\\}, wherehih\_\{i\}is the output hidden state for each token\. A pooling function converts these hidden states into a single representation followed byℓ2\\ell\_\{2\}normalization :hX=norm⁡\(Pool⁡\(Hθ​\(X\)\)\)h\_\{X\}=\\operatorname\{norm\}\(\\operatorname\{Pool\}\(H\_\{\\theta\}\(X\)\)\)\. The pooling function can be last\-token pooling\[[31](https://arxiv.org/html/2606.25674#bib.bib123)\], mean pooling\[[47](https://arxiv.org/html/2606.25674#bib.bib209)\], or architecture\-specific strategy\[[28](https://arxiv.org/html/2606.25674#bib.bib178)\]\. In this work, we append an\[EOS\]token to each input and use the hidden state of the final token as the text embedding\.

### 3\.2Low\-bit Embedding Backbone

BitEmbedis built by applying BitNet\-style quantization to the pretrained LLM backbone\. Following the BitNet line of work\[[62](https://arxiv.org/html/2606.25674#bib.bib199),[39](https://arxiv.org/html/2606.25674#bib.bib198)\], we quantize linear weights in transformer blocks to ternary values\. For a full\-precision weight matrixWW, the quantized weight is:

Qw​\(W\)=Δw⋅RoundClip⁡\(WΔw\+ϵ,−1,1\),Q\_\{w\}\(W\)=\\Delta\_\{w\}\\cdot\\operatorname\{RoundClip\}\(\\frac\{W\}\{\\Delta\_\{w\}\+\\epsilon\},\-1,1\),\(1\)whereΔw=mean⁡\(\|W\|\)\\Delta\_\{w\}=\\operatorname\{mean\}\(\|W\|\),RoundClip⁡\(X,a,b\)=max⁡\(a,min⁡\(b,round⁡\(X\)\)\)\\operatorname\{RoundClip\}\(X,a,b\)=\\operatorname\{max\}\(a,\\operatorname\{min\}\(b,\\operatorname\{round\}\(X\)\)\)\. This maps weights to scaled values in\{−Δw,0,Δw\}\\\{\-\\Delta\_\{w\},0,\\Delta\_\{w\}\\\}, yielding the 1\.58\-bit parameterization\.

For the activations entering each BitLinear layer, we apply token\-wise 8\-bit quantization\. Given an activation tensorXX, we computeγx=max⁡\|X\|\\gamma\_\{x\}=\\max\|X\|for each token and apply:

Qx​\(X\)=γx127​RoundClip⁡\(127γx\+ϵ​X,−128,127\)Q\_\{x\}\(X\)=\\frac\{\\gamma\_\{x\}\}\{127\}\\operatorname\{RoundClip\}\(\\frac\{127\}\{\\gamma\_\{x\}\+\\epsilon\}X,\-128,127\)\(2\)The quantizers in Equations[1](https://arxiv.org/html/2606.25674#S3.E1)and[2](https://arxiv.org/html/2606.25674#S3.E2)contain non\-differentiable operations, such as rounding and clipping\. Following prior low\-bit Transformer training practice\[[62](https://arxiv.org/html/2606.25674#bib.bib199),[40](https://arxiv.org/html/2606.25674#bib.bib197)\], we employ the straight\-through estimator \(STE\)\[[2](https://arxiv.org/html/2606.25674#bib.bib221)\]to approximate the gradients of these quantization operators during back\-propagation\.

Quantizing both weights and activations makes the transformer more sensitive to activation outliers and scale drift\[[40](https://arxiv.org/html/2606.25674#bib.bib197)\]\. Following prior works\[[39](https://arxiv.org/html/2606.25674#bib.bib198),[69](https://arxiv.org/html/2606.25674#bib.bib203)\], we perform model refinement by inserting sub\-layer normalization \(SubLN\) modules inside each transformer block\. Concretely, letAℓA\_\{\\ell\}be the concatenated multi\-head attention output before the output projection at layerℓ\\ell, and letFℓF\_\{\\ell\}be the intermediate feed\-forward representation before the down projection\. We refine the residual updates as

Yℓ\\displaystyle Y\_\{\\ell\}=Xℓ\+BitLinearo,ℓ⁡\(SubLN⁡\(Aℓ\)\),Xℓ\+1\\displaystyle=X\_\{\\ell\}\+\\operatorname\{BitLinear\}\_\{o,\\ell\}\\left\(\\operatorname\{SubLN\}\(A\_\{\\ell\}\)\\right\),X\_\{\\ell\+1\}=Yℓ\+BitLineard,ℓ⁡\(SubLN⁡\(Fℓ\)\),\\displaystyle=Y\_\{\\ell\}\+\\operatorname\{BitLinear\}\_\{d,\\ell\}\\left\(\\operatorname\{SubLN\}\(F\_\{\\ell\}\)\\right\),\(3\)whereXℓX\_\{\\ell\}denotes the input hidden states of layerℓ\\ell,YℓY\_\{\\ell\}denotes the intermediate state after the attention residual update, andBitLinear\\operatorname\{BitLinear\}denotes a linear projection computed with the quantized weights and activations above\. By normalizing these intermediate states locally, the quantized projections receive inputs with more consistent scales, which makes optimization less sensitive to low\-bit numerical noise\.

### 3\.3Continual Contrastive Pre\-training

Extreme quantization can distort the pretrained representation space and weaken the semantic knowledge encoded in the original LLM\. With such an extreme quantization constraint, relying only on supervised fine\-tuning data is insufficient for the model to rebuild a reliable semantic representation\. We therefore introduce continual contrastive pre\-training on large\-scale text pairs, which exposes the quantized model to broad semantic co\-occurrence and relevance signals before supervised training on the downstream task\. Given a training batchℬ=\{\(qi,pi\+\)\}i=1N\\mathcal\{B\}=\\\{\(q\_\{i\},p\_\{i\}^\{\+\}\)\\\}\_\{i=1\}^\{N\}, wherepi\+p\_\{i\}^\{\+\}is a positive text for queryqiq\_\{i\}, we feed them into the embedder to obtain embeddinghqih\_\{q\_\{i\}\}andhpi\+h\_\{p\_\{i\}^\{\+\}\}respectively\. Then we employ the standard InfoNCE\[[53](https://arxiv.org/html/2606.25674#bib.bib152)\]loss for optimization utilizing in\-batch negatives:

ℒcpt=−1N​∑i=1Nlog⁡exp⁡\(sθ​\(hqi,hpi\+\)/τ\)∑j=1Nexp⁡\(sθ​\(hqi,hpj\+\)/τ\)\.\\mathcal\{L\}\_\{\\mathrm\{cpt\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\(s\_\{\\theta\}\(h\_\{q\_\{i\}\},h\_\{p\_\{i\}^\{\+\}\}\)/\\tau\)\}\{\\sum\_\{j=1\}^\{N\}\\exp\(s\_\{\\theta\}\(h\_\{q\_\{i\}\},h\_\{p\_\{j\}^\{\+\}\}\)/\\tau\)\}\.\(4\)wheres​\(·\)s\(·\)measures the cosine similarity between two embedding vectors andτ\\tauis the temperature coefficient\. This stage provides broad contrastive adaptation and initializes a semantic embedding space before supervised fine\-tuning\.

### 3\.4Distillation\-based Supervised Fine\-tuning

After continual pre\-training, we train the low\-bit embedder on high\-quality supervised data\. A fine\-tuning batch is organized as\{\(Ii,qi,pi\+,\{pi,k−\}k=1K\)\}i=1N\\\{\(I\_\{i\},q\_\{i\},p\_\{i\}^\{\+\},\\\{p\_\{i,k\}^\{\-\}\\\}\_\{k=1\}^\{K\}\)\\\}\_\{i=1\}^\{N\}, where the queryqiq\_\{i\}is formatted with its task instructionIiI\_\{i\}andpi\+p\_\{i\}^\{\+\}and\{pi,k−\}k=1K\\\{p\_\{i,k\}^\{\-\}\\\}\_\{k=1\}^\{K\}represent the positive passage andKKhard negatives\. We optimize an InfoNCE loss with both in\-batch and mined hard negatives\. For each queryqiq\_\{i\}, the candidate set is𝒞i=\{pj\+\}j=1N∪\{pi,k−,1≤k≤K\}i=1N\\mathcal\{C\}\_\{i\}=\\\{p\_\{j\}^\{\+\}\\\}\_\{j=1\}^\{N\}\\cup\\\{p\_\{i,k\}^\{\-\},1\\leq k\\leq K\\\}\_\{i=1\}^\{N\}\. Given this candidate set, the supervised contrastive loss is

ℒctr=−1N​∑i=1Nlog⁡exp⁡\(sθ​\(hqi,hpi\+\)/τ\)∑c∈𝒞iexp⁡\(sθ​\(hqi,hc\)/τ\)\.\\mathcal\{L\}\_\{\\mathrm\{ctr\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\(s\_\{\\theta\}\(h\_\{q\_\{i\}\},h\_\{p\_\{i\}^\{\+\}\}\)/\\tau\)\}\{\\sum\_\{c\\in\\mathcal\{C\}\_\{i\}\}\\exp\(s\_\{\\theta\}\(h\_\{q\_\{i\}\},h\_\{c\}\)/\\tau\)\}\.\(5\)To further compensate for the representation loss introduced by extreme low\-bit quantization, we augment supervised fine\-tuning with teacher\-guided distillation\. The full\-precision teacher model is trained beforehand from the original LLM backbone on the supervised fine\-tuning data, and is then fixed to provide distillation signals\. We introduce two complementary distillation losses on top of the supervised contrastive objective\.

#### Similarity\-distribution distillation\.

In addition to the hard supervision used in the contrastive loss, we distill the teacher’s soft similarity distribution over the candidate set\. This distribution provides fine\-grained relative preference signals among positives and negatives candidates encoded by the teacher\. ForM∈\{T,S\}M\\in\\\{T,S\\\}, whereTTis the teacher andSSis the low\-bit embedder, define

PM​\(c∣qi\)=exp⁡\(sM​\(qi,c\)/τ\)∑c′∈𝒞iexp⁡\(sM​\(qi,c′\)/τ\)\.P\_\{M\}\(c\\mid q\_\{i\}\)=\\frac\{\\exp\(s\_\{M\}\(q\_\{i\},c\)/\\tau\)\}\{\\sum\_\{c^\{\\prime\}\\in\\mathcal\{C\}\_\{i\}\}\\exp\(s\_\{M\}\(q\_\{i\},c^\{\\prime\}\)/\\tau\)\}\.\(6\)The score\-level distillation loss minimizes the KL divergence from the teacher distribution to the low\-bit student model distribution:

ℒscore=1N∑i=1NDKL\(PT\(⋅∣qi\)∥PS\(⋅∣qi\)\)\.\\mathcal\{L\}\_\{\\mathrm\{score\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}D\_\{\\mathrm\{KL\}\}\\left\(P\_\{T\}\(\\cdot\\mid q\_\{i\}\)\\parallel P\_\{S\}\(\\cdot\\mid q\_\{i\}\)\\right\)\.\(7\)This encourages the embedding model to capture nuanced differences between the positive and negative learning from teacher model\.

#### Attention relation distillation\.

Score distillation aligns the final embedding similarities, but it does not directly constrain how the low\-bit encoder organizes token\-level interactions before pooling\. We therefore adopt a multi\-head attention distillation objective inspired by MiniLM\[[67](https://arxiv.org/html/2606.25674#bib.bib222),[66](https://arxiv.org/html/2606.25674#bib.bib223)\]\.

For a selected set of layersΩ\\Omegaand projection typesΦ=\{Q,K,V\}\\Phi=\\\{Q,K,V\\\}, letAℓ,ϕ,aM∈ℝL×dhA\_\{\\ell,\\phi,a\}^\{M\}\\in\\mathbb\{R\}^\{L\\times d\_\{h\}\}denote the normalized projected states of headaafrom modelM∈\{T,S\}M\\in\\\{T,S\\\}, whereLLis the sequence length anddhd\_\{h\}is the hidden dimension\.TTis the full\-precision teacher,SSis the low\-bit embedder, andNhN\_\{h\}is the number of heads\. For token positiont∈\[1,L\]t\\in\[1,L\], we compute its relation distribution to all positions as

Rℓ,ϕ,a,tM=softmax⁡\(Aℓ,ϕ,a,tM​\(Aℓ,ϕ,a,tM\)⊤dh\),R\_\{\\ell,\\phi,a,t\}^\{M\}=\\operatorname\{softmax\}\\left\(\\frac\{A\_\{\\ell,\\phi,a,t\}^\{M\}\(A\_\{\\ell,\\phi,a,t\}^\{M\}\)^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\),\(8\)The attention distillation loss averages the KL divergence between teacher and low\-bit relation distributions:

ℒattn=1\|Ω\|​\|Φ\|​∑ℓ∈Ω∑ϕ∈Φ1Nh​L​∑a=1Nh∑t=1LDKL​\(Rℓ,ϕ,a,tT∥Rℓ,ϕ,a,tS\)\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{attn\}\}=\\frac\{1\}\{\|\\Omega\|\|\\Phi\|\}\\sum\_\{\\ell\\in\\Omega\}\\sum\_\{\\phi\\in\\Phi\}\\frac\{1\}\{N\_\{h\}L\}\\sum\_\{a=1\}^\{N\_\{h\}\}\\sum\_\{t=1\}^\{L\}D\_\{\\mathrm\{KL\}\}\\left\(R\_\{\\ell,\\phi,a,t\}^\{T\}\\parallel R\_\{\\ell,\\phi,a,t\}^\{S\}\\right\)\.\(9\)Following MiniLM\[[67](https://arxiv.org/html/2606.25674#bib.bib222),[66](https://arxiv.org/html/2606.25674#bib.bib223)\], we perform attention relation distillation at only one layer\. This distillation objective encourages the low\-bit student model to retain the teacher’s fine\-grained structural dependencies\.

Finally, the distillation\-based supervised objective is:ℒdistill=ℒctr\+λs​ℒscore\+λa​ℒattn,\\mathcal\{L\}\_\{\\mathrm\{distill\}\}=\\mathcal\{L\}\_\{\\mathrm\{ctr\}\}\+\\lambda\_\{s\}\\mathcal\{L\}\_\{\\mathrm\{score\}\}\+\\lambda\_\{a\}\\mathcal\{L\}\_\{\\mathrm\{attn\}\},whereλs\\lambda\_\{s\}andλa\\lambda\_\{a\}are hyperparameters which balance score\-level and attention\-level distillation\.

### 3\.5Multi\-precision Embedding Quantization

In addition to quantizing the backbone, we train the output embeddings to remain robust under multiple storage precisions during the supervised fine\-tuning, which enables deployment under different memory budgets\. Lete∈ℝde\\in\\mathbb\{R\}^\{d\}denote the fp16 embedding produced byBitEmbed, and letℛ=\{1,2,4,8,16\}\\mathcal\{R\}=\\\{1,2,4,8,16\\\}denote the set of supported output bitwidths\. We use per\-vector absmax scaling,α​\(e\)=maxj⁡\|ej\|\\alpha\(e\)=\\max\_\{j\}\|e\_\{j\}\|, with a small lower bound for numerical stability\. The 16\-bit branch returns the original embedding unchanged, while lower\-bit settings store compact codes and use their reconstructed values for similarity computation during training\. For 1\-bit embeddings, we use scaled binary quantization:

zj\(1\)=sgn\+⁡\(ej\)∈\{−1,\+1\}\\displaystyle z^\{\(1\)\}\_\{j\}=\\operatorname\{sgn\}\_\{\+\}\(e\_\{j\}\)\\in\\\{\-1,\+1\\\}\(10\)wheree~j\(1\)=α​\(e\)​zj\(1\),sgn\+⁡\(0\)=1\\tilde\{e\}^\{\(1\)\}\_\{j\}=\\alpha\(e\)z^\{\(1\)\}\_\{j\},\\operatorname\{sgn\}\_\{\+\}\(0\)=1\. Forb∈\{2,4,8\}b\\in\\\{2,4,8\\\}, we use all2b2^\{b\}uniformly spaced levels in\[−α​\(e\),α​\(e\)\]\[\-\\alpha\(e\),\\alpha\(e\)\]\. Letmb=2b−1m\_\{b\}=2^\{b\}\-1\. Each dimension is quantized and dequantized as

zj\(b\)=clip⁡\(round⁡\(ej/α​\(e\)\+12​mb\),0,mb\)\\displaystyle z^\{\(b\)\}\_\{j\}=\\operatorname\{clip\}\\left\(\\operatorname\{round\}\\left\(\\frac\{e\_\{j\}/\\alpha\(e\)\+1\}\{2\}m\_\{b\}\\right\),0,m\_\{b\}\\right\)\(11\)wheree~j\(b\)=\(2​zj\(b\)mb−1\)​α​\(e\)\\tilde\{e\}^\{\(b\)\}\_\{j\}=\\left\(\\frac\{2z^\{\(b\)\}\_\{j\}\}\{m\_\{b\}\}\-1\\right\)\\alpha\(e\)\. Herez\(b\)z^\{\(b\)\}is the discrete code stored together with the scaleα​\(e\)\\alpha\(e\), whilee~\(b\)\\tilde\{e\}^\{\(b\)\}is the reconstructed vector used to compute similarities\. We use STE during training\.

Inspired by Matryoshka\-style multi\-precision training\[[50](https://arxiv.org/html/2606.25674#bib.bib204)\], we optimize theBitEmbedacross all supported output embedding precisions\. The final multi\-precision objective is

ℒmp=1\|ℛ\|​∑b∈ℛ\(ℒctr\(b\)\+λs​ℒscore\(b\)\)\+λa​ℒattn\.\\mathcal\{L\}\_\{\\mathrm\{mp\}\}=\\frac\{1\}\{\|\\mathcal\{R\}\|\}\\sum\_\{b\\in\\mathcal\{R\}\}\\left\(\\mathcal\{L\}\_\{\\mathrm\{ctr\}\}^\{\(b\)\}\+\\lambda\_\{s\}\\mathcal\{L\}\_\{\\mathrm\{score\}\}^\{\(b\)\}\\right\)\+\\lambda\_\{a\}\\mathcal\{L\}\_\{\\mathrm\{attn\}\}\.\(12\)

## 4Experiments

### 4\.1Experimental Setup

#### Backbones\.

We evaluate two LLM backbones: Qwen3\-0\.6B\[[73](https://arxiv.org/html/2606.25674#bib.bib191)\]and Gemma3\-270M\[[59](https://arxiv.org/html/2606.25674#bib.bib224)\]which differ in scales and architectures, to examine whether extreme low\-bit training can preserve the representation quality across different LLM\-based embedders\.

Table 1:Main results on MMTEB \(eng, v2\)\. We compareBitEmbedwith full\-precision teacher\. Cls\., Clust\., PairCls\., Rerank, Retr\., STS, Summ\., and Avg\. denote classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, summarization, and the overall average across all tasks, respectively\. “speed" denotes the token throughput of models on CPU with 8 threads\.
#### Training data\.

For continual pre\-training, following Multilingual E5\[[65](https://arxiv.org/html/2606.25674#bib.bib225)\], we trainBitEmbedon 1B text pairs\. For supervised fine\-tuning, we trainBitEmbedon the publicly available training data provided by BGE\-en\-ICL\[[31](https://arxiv.org/html/2606.25674#bib.bib123)\]\. More details on training data are available in Appendix[A](https://arxiv.org/html/2606.25674#A1)\. Specifically, during supervised fine\-tuning, the training data are organized in a retrieval\-style format, where each instance contains a query, a corresponding task instruction, a positive passage, and mined hard negatives\. To enable instruction\-aware embedding learning, we prepend the task instruction to the query side following the instruction\-aware training protocol commonly used by LLM\-based embedders:

qi​n​s​t=⟨Instruct⟩​\{task definition\}​⟨query⟩​\{q\}\\displaystyle q\_\{inst\}=\\texttt\{\\textlangle Instruct\\textrangle\}\\ \\\{\\texttt\{task\\ definition\}\\\}\\ \\texttt\{\\textlangle query\\textrangle\}\\ \\\{q\\\}\(13\)

#### Evaluation\.

We evaluateBitEmbedon Massive Multilingual Text Embedding Benchmark \(MMTEB\) \(eng, v2\)\[[17](https://arxiv.org/html/2606.25674#bib.bib130)\], which is an optimized version of MTEB\[[48](https://arxiv.org/html/2606.25674#bib.bib153)\]\. MMTEB covers diverse embedding tasks, including classification, clustering, pair classification, reranking, retrieval, semantic textual similarity \(STS\), and summarization\. Following the official metrics, we use accuracy for classification, V\-measure for clustering, average precision for pair classification, mean average precision \(MAP\) for reranking, nDCG@10 for retrieval, and Spearman correlation for STS and summarization\. Meanwhile we also evaluate the runtime efficiency of models which reports the token throughput \(tokens per second\) on CPU with 8 threads\. The evaluation instructions are provided in Appendix[B](https://arxiv.org/html/2606.25674#A2)\. Since our goal is to study performance\-preserving quantization for LLM\-based text embedding, we compare ourBitEmbedwith an FP16 model based on the same backbone which is fine\-tuned directly on the same supervised fine\-tuning data, rather than comparing against unrelated large\-scale embedding systems\.

#### Implementation details\.

We use a batch size of 128 for supervised fine\-tuning, and setτ\\tauin Eq[4](https://arxiv.org/html/2606.25674#S3.E4),[5](https://arxiv.org/html/2606.25674#S3.E5)and[6](https://arxiv.org/html/2606.25674#S3.E6)as 0\.02\. The number of hard negatives in supervised fine\-tuning data is set as 7\. For each backbone, the full\-precision teacher is trained on the supervised data using a learning rate of 5e\-6, and forBitEmbed, the learning rate of distillation\-based supervised fine\-tuning is set as 3e\-5\. The coefficient of similarity\-distribution distillation is set toλs=0\.2\\lambda\_\{s\}=0\.2, and the coefficient of multi\-head attention distillation is set toλa=104\\lambda\_\{a\}=10^\{4\}\. Following MiniLM\-style distillation\[[67](https://arxiv.org/html/2606.25674#bib.bib222)\], we distill attention relations from a single transformer layer, where we use layer 18 in the Qwen3\-0\.6B and layer 10 in the Gemma3 270M\.

### 4\.2Main Results

#### Overall performance\.

We compareBitEmbedwith the full precision teacher baselines on Qwen3\-0\.6B and Gemma3\-270M, and report the average score for each MMTEB \(eng, v2\) task category and the overall average score across all tasks in Table[1](https://arxiv.org/html/2606.25674#S4.T1)\. The detailed results on individual tasks are provided in the Appendix[C\.1](https://arxiv.org/html/2606.25674#A3.SS1)\. Furthermore, for the multilingual scenarios, we also conduct experiments and evaluate on multilingual MMTEB\. The experiment details are provided in Appendix[C\.2](https://arxiv.org/html/2606.25674#A3.SS2)\. Table[1](https://arxiv.org/html/2606.25674#S4.T1)shows that despite using an extreme low\-bit embedding backbone,BitEmbedremains largely comparable to the full\-precision teacher across both backbones\. On Qwen3\-0\.6B,BitEmbedachieves 67\.60, only 0\.35 points lower than the FP16 teacher\. On Gemma3\-270M,BitEmbedobtains 66\.10, closely matching the teacher score of 66\.71\. These small gaps indicate that BitNet\-style backbone conversion, when coupled with contrastive adaptation and teacher\-guided distillation, can recover most of the semantic representation quality required by text embedding tasks\.

Meanwhile,BitEmbedsubstantially improves inference efficiency\. On CPU with 8 threads, BITEMBED improves token throughput from 364\.36 to 830\.50 tokens/s on Qwen3\-0\.6B and from 1181\.28 to 2055\.47 tokens/s on Gemma3\-270M, achieving approximately 2× higher CPU token throughput on both backbones\. These results show that the practical utility ofBitEmbedin computationally limited scenarios\.

![Refer to caption](https://arxiv.org/html/2606.25674v1/x1.png)Figure 1:Performance\-precision trade\-off ofBitEmbedon Qwen3\-0\.6B and Gemma3\-270M\. We report the average MMTEB \(eng, v2\) performance of 1\-, 2\-, 4\-, 8\-, and 16\-bit output embeddings ofBitEmbed\.Table 2:Ablation study on Qwen3\-0\.6B with 16\-bit output embeddings\. CPT, SD, and AD denote continual pre\-training, similarity\-distribution distillation, and attention\-relation distillation, respectively\. BitNet SFT denotes supervised fine\-tuning after BitNet conversion, without continual pre\-training and distillation\.![Refer to caption](https://arxiv.org/html/2606.25674v1/x2.png)Figure 2:Task\-type sensitivity on MMTEB \(eng, v2\)\. Columns 1, 2, 4, and 8 report the performance differences relative to the 16\-bit embedding obtained fromBitEmbedunder lower precision\. TheBitEmbedcolumn reports performance differences betweenBitEmbedand the full\-precision teacher\.
#### Multi\-precision embedding performance\.

Figure[1](https://arxiv.org/html/2606.25674#S4.F1)further demonstrates the effect of output embedding precision\. With multi\-precision training, a singleBitEmbedcheckpoint supports multiple output embedding storage precisions and therefore offers a controllable trade\-off between embedding quality and vector\-index cost\. The 8\-bit and 4\-bit embeddings are nearly lossless compared with the 16\-bit branch on both backbones\. These results show that moderate output quantization can substantially reduce storage without materially changing downstream embedding quality\. Under more aggressive compression, the 1\-bit and 2\-bit output embeddings still preserve usable performance, achieving 64\.43 and 64\.90 on Qwen3\-0\.6B and 63\.01 and 63\.24 on Gemma3\-270M\. Although these settings introduce larger quantization noise, they reduce the storage by roughly16×16\\timesand8×8\\timescompared with full precision, making them attractive for storage\-constrained retrieval scenarios\. Overall, the multi\-precision training strategy enables practitioners to select an embedding precision according to deployment constraints while retaining most of the semantic utility of the original embedding\. Detailed experiments results for different embedding precisions are provided in Appendix[C\.3](https://arxiv.org/html/2606.25674#A3.SS3)\.

### 4\.3Ablation Study

We conduct ablations on Qwen3\-0\.6B to quantify the contribution of each component\. Table[2](https://arxiv.org/html/2606.25674#S4.T2)reports results with 16\-bit output embeddings on MMTEB \(eng, v2\)\. As shown, removing SubLN reduces the average score from 67\.60 to 65\.48, showing that the additional normalization modules are important for stabilizing extreme low\-bit embedding training\. Meanwhile, distillation also plays a critical role\. Removing attention\-relation and similarity\-distribution distillation respectively reduces the average score to 66\.49 and 66\.37, and removing both causes a larger drop to 64\.76, indicating that teacher guidance is important for recovering the representation quality of the low\-bit embedder\. Continual contrastive pre\-training is similarly critical\. Without it, the performance ofBitEmbeddrops to 64\.77, which demonstrates that continual pre\-training rebuilds a broad semantic space after extreme quantization\. Finally, directly fine\-tuning the converted BitNet\-style backbone without continual pre\-training and distillation achieves only 58\.92 average score, indicating that extreme low\-bit conversion substantially disrupts the embedding space and cannot be recovered by supervised fine\-tuning alone\.

### 4\.4Analysis

#### Task\-type sensitivity to quantization\.

To understand the task\-level sensitivity of the extreme low\-bitBitEmbed, we compareBitEmbedwith the full\-precision teacher across different task types\. As shown in the right column of Figure[2](https://arxiv.org/html/2606.25674#S4.F2), theBitEmbedremains close to the teacher on most task types, and even improves on some categories such as classification and summarization\. The remaining gaps are more substantial on retrieval, which is typically harder with encoding longer documents and further embedding matching\. Nevertheless,BitEmbedstill maintains strong performance, suggesting that BitNet\-style backbone quantization preserves most of the semantic structure needed by LLM\-based embedders\.

Then we analyze the sensitivity introduced by quantizing output embeddings ofBitEmbed\. The 1\-, 2\-, 4\-, and 8\-bit columns in Figure[2](https://arxiv.org/html/2606.25674#S4.F2)report differences relative to the original 16\-bit embedding obtained fromBitEmbed\. Retrieval is again the most sensitive task type\. In contrast, classification, pair classification, and STS are more stable\. This suggests that extreme low\-bit output embeddings may mainly affect the fine\-grained information required by retrieval, while still retaining sufficient semantic for tasks that rely on broader semantic separation\. Overall,BitEmbedpreserves overall semantic structure even at very low precision; with moderate low\-bit quantization, it can retain performance across all task types while substantially reducing embedding storage cost\.

![Refer to caption](https://arxiv.org/html/2606.25674v1/x3.png)Figure 3:Effect of the attention\-relation distillation layer selection\. Qwen3\-0\.6B is evaluated on even layers from 2 to 28, while Gemma3\-270M is evaluated on even layers from 2 to 18\.
![Refer to caption](https://arxiv.org/html/2606.25674v1/x4.png)Figure 4:Comparison betweenBitEmbedand MRL\. The1/X1/Xmeans that the storage budget is1/X1/Xof the original output embedding\. The correspondingBitEmbedprecisions are 1, 2, 4, 8, and 16 bits, while the corresponding MRL dimensions are 64, 128, 256, 512, and 1024\.

#### Effect of attention\-relation distillation layer

We further study the strategies for selecting the attention\-relation distillation layer\. Figure[4](https://arxiv.org/html/2606.25674#S4.F4)reports the average MMTEB \(eng, v2\) performance when distilling attention relations from different layers\. For Qwen3\-0\.6B, performance improves substantially from earlier layers to the middle\-later layers and peaks at layer 18\. For Gemma3\-270M, the best results are obtained around layers 10 and 12, while earlier layers are consistently weaker\. This trend is consistent across the two backbones: middle\-to\-later layers provide more effective distillation signals\. One possible explanation is that compared to earlier layers which mainly encode local lexical or syntactic patterns, middle\-to\-later layers better capture the semantic interactions that determine the final embedding space\. Very late layers do not always improve performance further, suggesting that the most useful relation information for embedding distillation lies in the semantically rich but not overly specialized part of the network\.

#### Effect of Multi\-precision Embedding Training

To explore the effect of multi\-precision embedding training, we further compare multi\-precision embedding training with a post\-hoc quantization baseline that directly quantizes embeddings without exposing the model to multiple precisions during training\. Table[3](https://arxiv.org/html/2606.25674#S4.T3)reports results on Qwen3\-0\.6B, and detailed results are provided in Appendix[C\.4](https://arxiv.org/html/2606.25674#A3.SS4)\. Multi\-precision training consistently improves performance, with larger gains at 1\-, 2\-, and 4\-bit settings\. The improvement is particularly visible on retrieval, where our method improves 1\-bit and 2\-bit embeddings by 2\.64 and 3\.10 points, respectively\. The comparison shows that by exposingBitEmbedto multiple embedding precisions during optimization,BitEmbedlearns embeddings whose similarity structure is more robust\.

Table 3:Effect of multi\-precision embedding training on Qwen3\-0\.6B\. “Post” denotes post\-hoc embedding quantization without multi\-precision training\.
#### Comparison with Matryoshka Representation Learning

We further compareBitEmbedwith Matryoshka Representation Learning \(MRL\)\[[26](https://arxiv.org/html/2606.25674#bib.bib205)\], which supports flexible storage by training nested prefix dimensions of the output embedding representation\. We conduct experiments on Qwen3\-0\.6B, whose output embedding dimension is 1024, and we set the MRL dimensions to 64, 128, 256, 512, and 1024, matching the storage costs of the 1\-, 2\-, 4\-, 8\-, and 16\-bit output embeddings ofBitEmbedrespectively for a fair comparison under the same embedding storage budgets\. As shown in Figure[4](https://arxiv.org/html/2606.25674#S4.F4),BitEmbedachieves better performance under all other storage budgets\. Although MRL is better at the 1/8 storage setting, outperforming BITEMBED by 0\.44, at the 1/16, 1/4, 1/2, and full\-storage settings,BitEmbedimproves the average MMTEB score over MRL by 1\.08, 1\.27, 0\.72, and 0\.30 points\. These results suggest that compared with MRL,BitEmbedcan keep the complete embedding space and learns to make it robust under low\-bit reconstruction, allowing more semantic factors to participate in similarity computation even when the storage budget is small\. Therefore, under the same storage budgets, our proposed multi\-precision embedding quantization provides a more effective trade\-off between embedding quality and storage efficiency\.

## 5Conclusion

In this paper, we presentedBitEmbed, an extreme low\-bit framework for LLM\-based text embeddings that jointly addresses encoding efficiency and embedding storage\.BitEmbedconverts pretrained LLM backbones into BitNet\-style embedders, adapting with continual contrastive pre\-training, and improves representation capabilities through supervised contrastive learning with similarity\-distribution and attention\-relation distillation\. FurthermoreBitEmbedsupports multi\-precision output embeddings, enabling a single model to serve different storage budgets\. Experiments on MMTEB \(eng, v2\) show thatBitEmbedremains close to full\-precision teachers, suggesting that extreme low\-bit training is a promising direction for efficient semantic representation\.

## References

- \[1\]E\. Agirre, D\. Cer, M\. Diab, and A\. Gonzalez\-Agirre\(2012\)Semeval\-2012 task 6: a pilot on semantic textual similarity\. in\* sem 2012: the first joint conference on lexical and computational semantics–volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation \(semeval 2012\)\.InProceedings of the Sixth International Workshop on Semantic Evaluation \(SemEval 2012\), Montréal, QC, Canada,pp\. 7–8\.Cited by:[5th item](https://arxiv.org/html/2606.25674#A1.I1.i5.p1.1)\.
- \[2\]Y\. Bengio, N\. Léonard, and A\. Courville\(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p2.3)\.
- \[3\]F\. Cai, T\. Chen, X\. Zhao, S\. Chen, H\. Zhang, S\. T\. Wu, I\. Gurevych, and H\. Koeppl\(2025\)Revela: dense retriever learning via language modeling\.arXiv preprint arXiv:2506\.16552\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[4\]I\. Casanueva, T\. Temčinas, D\. Gerz, M\. Henderson, and I\. Vulić\(2020\)Efficient intent detection with dual sentence encoders\.arXiv preprint arXiv:2003\.04807\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[5\]R\. Castro, A\. Panferov, R\. Tabesh, O\. Sieberling, J\. Chen, M\. Nikdan, S\. Ashkboos, and D\. Alistarh\(2026\)Quartet: native fp4 training can be optimal for large language models\.Advances in Neural Information Processing Systems38,pp\. 43552–43572\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1)\.
- \[6\]D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. Specia\(2017\)Semeval\-2017 task 1: semantic textual similarity\-multilingual and cross\-lingual focused evaluation\.arXiv preprint arXiv:1708\.00055\.Cited by:[5th item](https://arxiv.org/html/2606.25674#A1.I1.i5.p1.1)\.
- \[7\]D\. Chen and W\. Yih\(2020\)Open\-domain question answering\.InProceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts,pp\. 34–37\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[8\]H\. Chen, L\. Wang, N\. Yang, Y\. Zhu, Z\. Zhao, F\. Wei, and Z\. Dou\(2025\)Mme5: improving multimodal multilingual embeddings via high\-quality synthetic data\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 8254–8275\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[9\]M\. Chen, W\. Shao, P\. Xu, J\. Wang, P\. Gao, K\. Zhang, and P\. Luo\(2025\)Efficientqat: efficient quantization\-aware training for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10081–10100\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[10\]X\. Chen, A\. Zeynali, C\. Camargo, F\. Flöck, D\. Gaffney, P\. Grabowicz, S\. A\. Hale, D\. Jurgens, and M\. Samory\(2022\)SemEval\-2022 task 8: multilingual news article similarity\.InProceedings of the 16th International Workshop on Semantic Evaluation \(SemEval\-2022\),pp\. 1094–1106\.Cited by:[5th item](https://arxiv.org/html/2606.25674#A1.I1.i5.p1.1)\.
- \[11\]C\. Choi, J\. Kim, S\. Lee, J\. Kwon, S\. Gu, Y\. Kim, M\. Cho, and J\. Sohn\(2024\)Linq\-embed\-mistral technical report\.arXiv preprint arXiv:2412\.03223\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[12\]A\. Cohan, S\. Feldman, I\. Beltagy, D\. Downey, and D\. S\. Weld\(2020\)Specter: document\-level representation learning using citation\-informed transformers\.arXiv preprint arXiv:2004\.07180\.Cited by:[2nd item](https://arxiv.org/html/2606.25674#A1.I1.i2.p1.1)\.
- \[13\]DataCanary, hilfialkaff, L\. Jiang, M\. Risdal, N\. Dandekar, and tomtung\(2017\)Quora question pairs\.Note:[https://kaggle\.com/competitions/quora\-question\-pairs](https://kaggle.com/competitions/quora-question-pairs)KaggleCited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[14\]T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer\(2022\)Gpt3\. int8 \(\): 8\-bit matrix multiplication for transformers at scale\.Advances in neural information processing systems35,pp\. 30318–30332\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[15\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\-06\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[16\]D\. Du, Y\. Zhang, S\. Cao, J\. Guo, T\. Cao, X\. Chu, and N\. Xu\(2024\)Bitdistiller: unleashing the potential of sub\-4\-bit llms via self\-distillation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 102–116\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[17\]K\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzemiński, G\. I\. Winata,et al\.\(2025\)Mmteb: massive multilingual text embedding benchmark\.arXiv preprint arXiv:2502\.13595\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1),[§1](https://arxiv.org/html/2606.25674#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px3.p1.1)\.
- \[18\]A\. Fan, Y\. Jernite, E\. Perez, D\. Grangier, J\. Weston, and M\. Auli\(2019\)ELI5: long form question answering\.arXiv preprint arXiv:1907\.09190\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[19\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[20\]T\. Gao, X\. Yao, and D\. Chen\(2021\)Simcse: simple contrastive learning of sentence embeddings\.arXiv preprint arXiv:2104\.08821\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[21\]R\. Gong, Y\. Ding, Z\. Wang, C\. Lv, X\. Zheng, J\. Du, Y\. Yong, S\. Gu, H\. Qin, J\. Guo,et al\.\(2025\)A survey of low\-bit large language models: basics, systems, and algorithms\.Neural networks,pp\. 107856\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1)\.
- \[22\]F\. Huang, F\. Wu, Z\. Zhang, Q\. Wang, L\. Zhang, G\. M\. Boquet, and H\. Chen\(2025\)GeoGPT\. rag technical report\.arXiv preprint arXiv:2509\.09686\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[23\]H\. Huang and D\. Wu\(2025\)Quaff: quantized parameter\-efficient fine\-tuning under outlier spatial stability hypothesis\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6481–6496\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1)\.
- \[24\]V\. Karpukhin, B\. Oguz, S\. Min, P\. S\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih\(2020\)Dense passage retrieval for open\-domain question answering\.\.InEMNLP \(1\),pp\. 6769–6781\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[25\]O\. Khattab and M\. Zaharia\(2020\)Colbert: efficient and effective passage search via contextualized late interaction over bert\.InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval,pp\. 39–48\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[26\]A\. Kusupati, G\. Bhatt, A\. Rege, M\. Wallingford, A\. Sinha, V\. Ramanujan, W\. Howard\-Snyder, K\. Chen, S\. Kakade, P\. Jain,et al\.\(2022\)Matryoshka representation learning\.Advances in Neural Information Processing Systems35,pp\. 30233–30249\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p4.1),[§4\.4](https://arxiv.org/html/2606.25674#S4.SS4.SSS0.Px4.p1.1)\.
- \[27\]K\. Lang\(1995\)Newsweeder: learning to filter netnews\.InMachine learning proceedings 1995,pp\. 331–339\.Cited by:[4th item](https://arxiv.org/html/2606.25674#A1.I1.i4.p1.1)\.
- \[28\]C\. Lee, R\. Roy, M\. Xu, J\. Raiman, M\. Shoeybi, B\. Catanzaro, and W\. Ping\(2024\)Nv\-embed: improved techniques for training llms as generalist embedding models\.arXiv preprint arXiv:2405\.17428\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.25674#S3.SS1.p1.9)\.
- \[29\]J\. Lee, Z\. Dai, X\. Ren, B\. Chen, D\. Cer, J\. R\. Cole, K\. Hui, M\. Boratko, R\. Kapadia, W\. Ding,et al\.\(2024\)Gecko: versatile text embeddings distilled from large language models\.arXiv preprint arXiv:2403\.20327\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[30\]C\. Li, Z\. Liu, S\. Xiao, Y\. Shao, and D\. Lian\(2024\)Llama2vec: unsupervised adaptation of large language models for dense retrieval\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3490–3500\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[31\]C\. Li, M\. Qin, S\. Xiao, J\. Chen, K\. Luo, Y\. Shao, D\. Lian, and Z\. Liu\(2024\)Making text embedders few\-shot learners\.arXiv preprint arXiv:2409\.15700\.Cited by:[Appendix A](https://arxiv.org/html/2606.25674#A1.p1.1),[§3\.1](https://arxiv.org/html/2606.25674#S3.SS1.p1.9),[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px2.p1.1)\.
- \[32\]H\. Li, A\. Arora, S\. Chen, A\. Gupta, S\. Gupta, and Y\. Mehdad\(2020\)MTOP: a comprehensive multilingual task\-oriented semantic parsing benchmark\.arXiv preprint arXiv:2008\.09335\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[33\]Z\. Li, X\. Zhang, Y\. Zhang, D\. Long, P\. Xie, and M\. Zhang\(2023\)Towards general text embeddings with multi\-stage contrastive learning\.arXiv preprint arXiv:2308\.03281\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[34\]Z\. Li, Y\. Su, R\. Yang, C\. Xie, Z\. Wang, Z\. Xie, N\. Wong, and H\. Yang\(2025\)Quantization meets reasoning: exploring llm low\-bit quantization degradation for mathematical reasoning\.arXiv preprint arXiv:2501\.03035\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1)\.
- \[35\]Z\. Li, C\. Tao, J\. Feng, T\. Shen, D\. Zhao, X\. Geng, and D\. Jiang\(2023\-07\)FAA: fine\-grained attention alignment for cascade document ranking\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1688–1700\.External Links:[Link](https://aclanthology.org/2023.acl-long.94/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.94)Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[36\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[37\]X\. Liu, C\. Wang, Y\. Leng, and C\. Zhai\(2018\)Linkso: a dataset for learning to retrieve similar question answer pairs on software development forums\.InProceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering,pp\. 2–5\.Cited by:[2nd item](https://arxiv.org/html/2606.25674#A1.I1.i2.p1.1)\.
- \[38\]Z\. Liu, B\. Oguz, C\. Zhao, E\. Chang, P\. Stock, Y\. Mehdad, Y\. Shi, R\. Krishnamoorthi, and V\. Chandra\(2024\)Llm\-qat: data\-free quantization aware training for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 467–484\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[39\]S\. Ma, H\. Wang, S\. Huang, X\. Zhang, Y\. Hu, T\. Song, Y\. Xia, and F\. Wei\(2025\)BitNet b1\. 58 2b4t technical report\.arXiv preprint arXiv:2504\.12285\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p3.3)\.
- \[40\]S\. Ma, H\. Wang, L\. Ma, L\. Wang, W\. Wang, S\. Huang, L\. Dong, R\. Wang, J\. Xue, and F\. Wei\(2024\)The era of 1\-bit llms: all large language models are in 1\.58 bits\.arXiv preprint arXiv:2402\.17764\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p2.3),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p3.3)\.
- \[41\]X\. Ma, L\. Wang, N\. Yang, F\. Wei, and J\. Lin\(2024\)Fine\-tuning llama for multi\-stage text retrieval\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2421–2425\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[42\]A\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. Potts\(2011\)Learning word vectors for sentiment analysis\.InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies,pp\. 142–150\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[43\]Maggie, P\. Culliton, and W\. Chen\(2020\)Tweet sentiment extraction\.Note:[https://kaggle\.com/competitions/tweet\-sentiment\-extraction](https://kaggle.com/competitions/tweet-sentiment-extraction)KaggleCited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[44\]M\. Maia, S\. Handschuh, A\. Freitas, B\. Davis, R\. McDermott, M\. Zarrouk, and A\. Balahur\(2018\)Www’18 open challenge: financial opinion mining and question answering\.InCompanion proceedings of the the web conference 2018,pp\. 1941–1942\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[45\]J\. McAuley and J\. Leskovec\(2013\)Hidden factors and hidden topics: understanding rating dimensions with review text\.InProceedings of the 7th ACM conference on Recommender systems,pp\. 165–172\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[46\]R\. Meng, Y\. Liu, S\. R\. Joty, C\. Xiong, Y\. Zhou, and S\. Yavuz\(2024\)Sfrembedding\-mistral: enhance text retrieval with transfer learning\.Salesforce AI Research Blog3,pp\. 6\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[47\]N\. Muennighoff, H\. Su, L\. Wang, N\. Yang, F\. Wei, T\. Yu, A\. Singh, and D\. Kiela\(2025\)Generative representational instruction tuning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 45544–45613\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.25674#S3.SS1.p1.9)\.
- \[48\]N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers\(2023\-05\)MTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 2014–2037\.External Links:[Link](https://aclanthology.org/2023.eacl-main.148/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.148)Cited by:[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px3.p1.1)\.
- \[49\]N\. Muennighoff\(2022\)Sgpt: gpt sentence embeddings for semantic search\.arXiv preprint arXiv:2202\.08904\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[50\]P\. Nair, P\. Datta, J\. Dean, P\. Jain, and A\. Kusupati\(2025\)Matryoshka quantization\.arXiv preprint arXiv:2502\.06786\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p4.1),[§3\.5](https://arxiv.org/html/2606.25674#S3.SS5.p2.1)\.
- \[51\]T\. Nguyen, M\. Rosenberg, X\. Song, J\. Gao, S\. Tiwary, R\. Majumder, and L\. Deng\(2016\)Ms marco: a human\-generated machine reading comprehension dataset\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[52\]J\. O’Neill, P\. Rozenshtein, R\. Kiryo, M\. Kubota, and D\. Bollegala\(2021\)I wish i would have loved this one, but i didn’t–a multilingual dataset for counterfactual detection in product reviews\.arXiv preprint arXiv:2104\.06893\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[53\]A\. v\. d\. Oord, Y\. Li, and O\. Vinyals\(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§3\.3](https://arxiv.org/html/2606.25674#S3.SS3.p1.5)\.
- \[54\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 3982–3992\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[55\]E\. Saravia, H\. T\. Liu, Y\. Huang, J\. Wu, and Y\. Chen\(2018\)CARER: contextualized affect representations for emotion recognition\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 3687–3697\.Cited by:[3rd item](https://arxiv.org/html/2606.25674#A1.I1.i3.p1.1)\.
- \[56\]A\. Sorokin, N\. Buzun, A\. Anokhin, O\. Inozemcev, E\. Vedernikov, P\. Anokhin, M\. Burtsev, T\. Alexey, Y\. Wenshuai, and E\. Burnaev\(2025\)Q\-rag: long context multi\-step retrieval via value\-based embedder training\.arXiv preprint arXiv:2511\.07328\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[57\]H\. Su, W\. Shi, J\. Kasai, Y\. Wang, Y\. Hu, M\. Ostendorf, W\. Yih, N\. A\. Smith, L\. Zettlemoyer, and T\. Yu\(2023\-07\)One embedder, any task: instruction\-finetuned text embeddings\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1102–1121\.External Links:[Link](https://aclanthology.org/2023.findings-acl.71/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.71)Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[58\]C\. Tao, T\. Shen, S\. Gao, J\. Zhang, Z\. Li, K\. Hua, W\. Hu, Z\. Tao, and S\. Ma\(2024\)Llms are also effective embedding models: an in\-depth overview\.arXiv preprint arXiv:2412\.12591\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p4.1),[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[59\]G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px1.p1.1)\.
- \[60\]J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal\(2018\)FEVER: a large\-scale dataset for fact extraction and verification\.arXiv preprint arXiv:1803\.05355\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[61\]H\. Wachsmuth, S\. Syed, and B\. Stein\(2018\)Retrieval of the best counterargument without prior topic knowledge\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 241–251\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[62\]H\. Wang, S\. Ma, L\. Dong, S\. Huang, H\. Wang, L\. Ma, F\. Yang, R\. Wang, Y\. Wu, and F\. Wei\(2023\)Bitnet: scaling 1\-bit transformers for large language models\.arXiv preprint arXiv:2310\.11453\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p2.3)\.
- \[63\]L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. Wei\(2022\)Text embeddings by weakly\-supervised contrastive pre\-training\.arXiv preprint arXiv:2212\.03533\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[64\]L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei\(2024\-08\)Improving text embeddings with large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11897–11916\.External Links:[Link](https://aclanthology.org/2024.acl-long.642/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.642)Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[65\]L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei\(2024\)Multilingual e5 text embeddings: a technical report\.arXiv preprint arXiv:2402\.05672\.Cited by:[§C\.2](https://arxiv.org/html/2606.25674#A3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px2.p1.1)\.
- \[66\]W\. Wang, H\. Bao, S\. Huang, L\. Dong, and F\. Wei\(2021\)Minilmv2: multi\-head self\-attention relation distillation for compressing pretrained transformers\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 2140–2151\.Cited by:[§3\.4](https://arxiv.org/html/2606.25674#S3.SS4.SSS0.Px2.p1.1),[§3\.4](https://arxiv.org/html/2606.25674#S3.SS4.SSS0.Px2.p2.13)\.
- \[67\]W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou\(2020\)Minilm: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.Advances in neural information processing systems33,pp\. 5776–5788\.Cited by:[§3\.4](https://arxiv.org/html/2606.25674#S3.SS4.SSS0.Px2.p1.1),[§3\.4](https://arxiv.org/html/2606.25674#S3.SS4.SSS0.Px2.p2.13),[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px4.p1.3)\.
- \[68\]Y\. Wang, H\. Wang, and S\. Q\. Zhang\(2026\)Qsvd: efficient low\-rank approximation for unified query\-key\-value weight compression in low\-precision vision\-language models\.Advances in Neural Information Processing Systems38,pp\. 1789–1820\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1)\.
- \[69\]X\. Wu, S\. Huang, W\. Wang, T\. Song, L\. Dong, Y\. Xia, and F\. Wei\(2025\)BitNet distillation\.arXiv preprint arXiv:2510\.13998\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.25674#S3.SS2.p3.3)\.
- \[70\]G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han\(2023\)Smoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[71\]L\. Xiong, C\. Xiong, Y\. Li, K\. Tang, J\. Liu, P\. Bennett, J\. Ahmed, and A\. Overwijk\(2020\)Approximate nearest neighbor negative contrastive learning for dense text retrieval\.arXiv preprint arXiv:2007\.00808\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[72\]Y\. Xu, X\. Han, Z\. Yang, S\. Wang, Q\. Zhu, Z\. Liu, W\. Liu, and W\. Che\(2024\)Onebit: towards extremely low\-bit large language models\.Advances in Neural Information Processing Systems37,pp\. 66357–66382\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[73\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2606.25674#S4.SS1.SSS0.Px1.p1.1)\.
- \[74\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.arXiv preprint arXiv:1809\.09600\.Cited by:[1st item](https://arxiv.org/html/2606.25674#A1.I1.i1.p1.1)\.
- \[75\]Z\. Yuan, L\. Niu, J\. Liu, W\. Liu, X\. Wang, Y\. Shang, G\. Sun, Q\. Wu, J\. Wu, and B\. Wu\(2023\)Rptq: reorder\-based post\-training quantization for large language models\.arXiv preprint arXiv:2304\.01089\.Cited by:[§2\.2](https://arxiv.org/html/2606.25674#S2.SS2.p1.1)\.
- \[76\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.
- \[77\]P\. Zhao, H\. Zhang, Q\. Yu, Z\. Wang, Y\. Geng, F\. Fu, L\. Yang, W\. Zhang, J\. Jiang, and B\. Cui\(2024\)Retrieval\-augmented generation for ai\-generated content: a survey\.arXiv preprint arXiv:2402\.19473\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[78\]W\. X\. Zhao, J\. Liu, R\. Ren, and J\. Wen\(2024\)Dense text retrieval based on pretrained language models: a survey\.ACM Transactions on Information Systems42\(4\),pp\. 1–60\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[79\]X\. Zhao, M\. Wang, X\. Zhao, J\. Li, S\. Zhou, D\. Yin, Q\. Li, J\. Tang, and R\. Guo\(2023\)Embedding in recommender systems: a survey\.arXiv preprint arXiv:2310\.18608\.Cited by:[§1](https://arxiv.org/html/2606.25674#S1.p1.1)\.
- \[80\]X\. Zhao, X\. Hu, Z\. Shan, S\. Huang, Y\. Zhou, X\. Zhang, Z\. Sun, Z\. Liu, D\. Li, X\. Wei,et al\.\(2025\)Kalm\-embedding\-v2: superior training techniques and data inspire a versatile embedding model\.arXiv preprint arXiv:2506\.20923\.Cited by:[§2\.1](https://arxiv.org/html/2606.25674#S2.SS1.p1.1)\.

## Appendix ATraining Data

Following BGE\-en\-ICL\[[31](https://arxiv.org/html/2606.25674#bib.bib123)\], our training data contains retrieval, reranking, classification, clustering, and STS tasks:

- •*Retrieval:*ELI5\[[18](https://arxiv.org/html/2606.25674#bib.bib154)\], HotpotQA\[[74](https://arxiv.org/html/2606.25674#bib.bib155)\], FEVER\[[60](https://arxiv.org/html/2606.25674#bib.bib156)\], MSMARCO passage and document ranking\[[51](https://arxiv.org/html/2606.25674#bib.bib108)\], NQ\[[24](https://arxiv.org/html/2606.25674#bib.bib109)\], NLI\[[20](https://arxiv.org/html/2606.25674#bib.bib157)\], SQuAD\[[24](https://arxiv.org/html/2606.25674#bib.bib109)\], TriviaQA\[[24](https://arxiv.org/html/2606.25674#bib.bib109)\], Quora Duplicate Questions\[[13](https://arxiv.org/html/2606.25674#bib.bib158)\], Arguana\[[61](https://arxiv.org/html/2606.25674#bib.bib159)\], FiQA\[[44](https://arxiv.org/html/2606.25674#bib.bib160)\]\.
- •*Reranking:*SciDocsRR\[[12](https://arxiv.org/html/2606.25674#bib.bib161)\], StackOverFlowDupQuestions\[[37](https://arxiv.org/html/2606.25674#bib.bib162)\]\.
- •*Classification:*AmazonReviewsClassification\[[45](https://arxiv.org/html/2606.25674#bib.bib163)\], AmazonCounterfactualClassification\[[52](https://arxiv.org/html/2606.25674#bib.bib164)\], Banking77Classification\[[4](https://arxiv.org/html/2606.25674#bib.bib165)\], EmotionClassification\[[55](https://arxiv.org/html/2606.25674#bib.bib166)\], TweetSentimentExtractionClassification\[[43](https://arxiv.org/html/2606.25674#bib.bib168)\], MTOPIntentClassification\[[32](https://arxiv.org/html/2606.25674#bib.bib167)\], IMDBClassification\[[42](https://arxiv.org/html/2606.25674#bib.bib169)\], ToxicConversationsClassification\[[17](https://arxiv.org/html/2606.25674#bib.bib130)\]\.
- •*Clustering:*Arxiv / Biorxiv / Medrxiv / Reddit / StackExchange Clustering S2S/P2P, TwentyNewsgroupsClustering\[[27](https://arxiv.org/html/2606.25674#bib.bib170)\]\.
- •*STS:*STS12\[[1](https://arxiv.org/html/2606.25674#bib.bib172)\], STS22\[[10](https://arxiv.org/html/2606.25674#bib.bib173)\], STS\-Benchmark\[[6](https://arxiv.org/html/2606.25674#bib.bib174)\]\.

Table 4:Task instructions for MMTEB\.Table 5:Detail results in MMTEB \(eng, v2\)\.Table 6:Performance ofBitEmbedon multilingual MMTEB based on Qwen3\-0\.6B and Gemma3\-270M\.Table 7:Performance on MMTEB \(eng, v2\) across multi\-precision output embeddings ofBitEmbed\. The “\-bbbit" denotes the output embeddings ofBitEmbedare stored and scored atbb\-bit precision\.Table 8:Performance on MMTEB \(eng, v2\) across multi\-precision output embeddings of post\-hoc quantization baseline whose baskbone model is Qwen3\-0\.6B\. The “\-bbbit" denotes the output embeddings of baseline are stored and scored atbb\-bit precision\.The total training set includes 2\.1 million samples\.

## Appendix BEvaluation Details

The instruction for evaluation on MMTEB \(eng, v2\) is shown in Table[4](https://arxiv.org/html/2606.25674#A1.T4)\.

## Appendix CPerformance on MMTEB

### C\.1Performance Details on MMTEB \(eng, v2\)

The experiments results ofBitEmbedand teacher models based on Qwen3\-0\.6B and Gemma3\-270M across all tasks on MMTEB \(eng, v2\) are available in Table[5](https://arxiv.org/html/2606.25674#A1.T5)\.

### C\.2Performance on Multilingual MMTEB

We demonstrate the performance on multilingual MMTEB ofBitEmbedbased on Qwen3\-0\.6B and Gemma3\-270M in Table[6](https://arxiv.org/html/2606.25674#A1.T6)\. In multilingual scenarios, we conduct BitNet\-style quantization based on the backbone models, and conduct continual pre\-training following multilingual E5\[[65](https://arxiv.org/html/2606.25674#bib.bib225)\]\. Then we conduct distillation\-based superivised fine\-tuning utilizing the traning data same as harrier\-oss\-v1111https://huggingface\.co/microsoft/harrier\-oss\-v1\-0\.6b\.

### C\.3Multi\-precision embedding performance on MMTEB \(eng, v2\)

We demonstrate the performance on MMTEB \(eng, v2\) ofBitEmbedwith multi\-precision output embeddings in Table[7](https://arxiv.org/html/2606.25674#A1.T7)\.

### C\.4Performance of post\-hoc quantization baselines on MMTEB \(eng, v2\)

We demonstrate the performance on MMTEB \(eng, v2\) of post\-hoc quantization baseline with multi\-precision output embeddings in Table[8](https://arxiv.org/html/2606.25674#A1.T8)\.