The Embedder's Dilemma: LLMs Are Better, but at What Cost?

# LLMs Are Better, but at What Cost?
Source: [https://arxiv.org/html/2608.12875](https://arxiv.org/html/2608.12875)
## The Embedder’s Dilemma: LLMs Are Better, but at What Cost?

Niklas MuennighoffAffiliation:Stanford UniversityJinhyuk LeeAffiliation:Independent Researcher

###### Abstract

Should you replace your text\-embedding pipeline with a large language model? We answer this with a controlled, cost\-aware comparison of ten LLMs across six families and 26 embedding models \(118M to 14B parameters\) on 37 tasks spanning classification, semantic textual similarity \(STS\), clustering, pair classification, and retrieval\. In aggregate the two paradigms are effectively tied: the best LLM \(Gemini 3\.1 Pro, 77\.6\) and the best embedding model \(77\.2\) differ by 0\.4 points\. Their strengths differ by task: LLMs lead on reasoning\-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification\. Reaching that parity is expensive\. An LLM costs up to 1,431×\\timesmore than an embedding model of comparable quality \($154 vs\. $0\.11 per benchmark pass\), and the open LLMs tested process tokens 2\.5 to 736×\\timesmore slowly on the same GPU\. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation\. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3\.1 Pro\. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning\-intensive retrieval\. Our code, datasets, and results are publicly available at[https://github\.com/embeddings\-benchmark/embedders\-dilemma](https://github.com/embeddings-benchmark/embedders-dilemma)\.

Figure 1:Cost vs\. performance across 36 models on MTEB\(LLM\)\.The frontier contains the leading embedding models and Gemini 3\.1 Pro, which extends it by 0\.4 points at 1,431×\\timesthe cost of a comparable embedding\.## 1Introduction

We show that an LLM without task\-specific training now matches the best text embedding models across a broad suite of standard embedding tasks\. Embedding models reach this quality through specialised contrastive training, hard\-negative mining, and multi\-stage distillation; our best LLM reaches it without a dedicated embedding\-training pipeline\.111While embedding models have increasingly adopted LLMs as their backbone\([67](https://arxiv.org/html/2608.12875#bib.bib7);[51](https://arxiv.org/html/2608.12875#bib.bib6)\), they output vectors and use training tailored to embedding tasks\. Meanwhile, “LLMs” in this work are generative models prompted to output text, with no dedicated embedding training\.The deployment costs differ sharply: LLM inference is substantially more expensive and slower on the same hardware\. We call this theEmbedder’s Dilemma: LLMs now match the best embedding models in aggregate, but at much higher cost and lower throughput\.

We ask a practical question:*should you replace your embedding pipeline with an LLM?*Our results favour embedding models as the default and LLMs for reasoning\-heavy retrieval\. A hybrid pipeline captures the retrieval benefit while controlling cost and throughput\.

We compare ten frontier LLMs from six families with 26 text embedding models \(118M–14B parameters\)\. The evaluation usesMTEB\(LLM\), a 37\-task benchmark built from the Massive Text Embedding Benchmark\([49](https://arxiv.org/html/2608.12875#bib.bib10), MTEB;\), covering classification, semantic textual similarity \(STS\), clustering, pair classification, and retrieval\. The full model list appears in §[3](https://arxiv.org/html/2608.12875#S3)\. Each MTEB\(LLM\) task is a fixed subset of the corresponding MTEB task, released on Hugging Face undermteb/llm\-eval\-\*; both paradigms are scored on exactly that subset\. MTEB\(LLM\) scores compare models within this benchmark; MTEB leaderboard scores use the full test sets and follow a different evaluation basis\. We pair the quality comparison with exact cost accounting \(API token tracking for LLMs; GPU throughput benchmarking for embedding models\), producing a cost–performance Pareto frontier spanning both paradigms \(Figure[1](https://arxiv.org/html/2608.12875#S0.F1)\)\.

Contributions\.We release MTEB\(LLM\), the first benchmark to compare LLMs and embedding models across all five MTEB task categories, with cost measured alongside quality for every model\. We implement it with the MTEB framework, part of an ecosystem that spans multilingual text \(MMTEB;[23](https://arxiv.org/html/2608.12875#bib.bib15)\), image \(MIEB;[75](https://arxiv.org/html/2608.12875#bib.bib12)\), audio \(MAEB;[21](https://arxiv.org/html/2608.12875#bib.bib13)\), and video \(MVEB;[22](https://arxiv.org/html/2608.12875#bib.bib14)\)\. The Pareto frontier contains the leading embeddings and Gemini 3\.1 Pro, whose small score gain comes at three orders of magnitude higher cost\. We identify the*thinking\-token tax*: reasoning accounts for 28–81% of LLM inference cost, while lower reasoning budgets preserve retrieval quality for most models in our ablation \(§[4\.6](https://arxiv.org/html/2608.12875#S4.SS6)\)\.

Our empirical findings are:

1. 1\.In aggregate, the paradigms tie\.The best LLM and embedding model differ by 0\.4 points, within statistical noise \(§[4\.2](https://arxiv.org/html/2608.12875#S4.SS2)\)\. The pattern holds across the task suite\.
2. 2\.The advantage is task\-specific\.LLMs lead on reasoning\-heavy retrieval, embedding models lead on classification, and the paradigms are statistically even on clustering, STS, and pair classification\. LLMs excel when cross\-document reasoning is required; embedding pipelines excel wherever geometric similarity or labelled\-reference matching suffices \(§[3\.2](https://arxiv.org/html/2608.12875#S3.SS2)–[4\.2](https://arxiv.org/html/2608.12875#S4.SS2)\)\.
3. 3\.The cost gap is large\.Gemini 3\.1 Pro costs 1,431×\\timesas much as a comparable embedding model, and the gap persists across hardware scenarios\. Reasoning contributes substantially to this asymmetry\. Lower reasoning budgets preserve or improve retrieval for most evaluated models \(§[4\.4](https://arxiv.org/html/2608.12875#S4.SS4)–[4\.6](https://arxiv.org/html/2608.12875#S4.SS6)\)\.
4. 4\.Throughput lags by orders of magnitude, even on the same hardware\.Served on an identical H100, open\-weight LLMs process 2\.5–736×\\timesfewer tokens per second than embedding models, consistent with the architectural difference between autoregressive generation and a single encoder pass \(§[4\.5](https://arxiv.org/html/2608.12875#S4.SS5); Figure[7](https://arxiv.org/html/2608.12875#A4.F7)\)\.

LLMs and embedding models therefore serve complementary roles\. Embedding pipelines remain the efficient default for classification, similarity, and clustering; LLMs are most useful for reasoning\-intensive retrieval\.

## 2Related Work

### 2\.1Text Embedding Models, Evaluation, and Cost

Dense text representations have evolved rapidly from contextualised encoders\([18](https://arxiv.org/html/2608.12875#bib.bib34);[55](https://arxiv.org/html/2608.12875#bib.bib16)\)to billion\-parameter, instruction\-tuned models\([62](https://arxiv.org/html/2608.12875#bib.bib42)\)\. Leading systems such as E5\([71](https://arxiv.org/html/2608.12875#bib.bib17);[72](https://arxiv.org/html/2608.12875#bib.bib18)\), NV\-Embed\([37](https://arxiv.org/html/2608.12875#bib.bib54)\), SFR\-2\([47](https://arxiv.org/html/2608.12875#bib.bib39)\), and Gemini Embedding\([39](https://arxiv.org/html/2608.12875#bib.bib56)\)achieve strong performance through contrastive training on large curated corpora\. MTEB\([49](https://arxiv.org/html/2608.12875#bib.bib10)\)standardised evaluation across eight task categories and now tracks hundreds of models on a public leaderboard\.222[https://huggingface\.co/spaces/mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard)The framework has expanded across multilingual text \(MMTEB;[23](https://arxiv.org/html/2608.12875#bib.bib15)\), image \(MIEB;[75](https://arxiv.org/html/2608.12875#bib.bib12)\), audio \(MAEB;[21](https://arxiv.org/html/2608.12875#bib.bib13)\), and video \(MVEB;[22](https://arxiv.org/html/2608.12875#bib.bib14)\)\. HUME\([5](https://arxiv.org/html/2608.12875#bib.bib11)\)measures human performance on MTEB tasks and benchmarks LLMs as annotators\. Our study asks whether an LLM can replace an embedding pipeline\. We extend this line of evaluation with exact cost and throughput accounting across both paradigms\.

[56](https://arxiv.org/html/2608.12875#bib.bib29)introduced “Green AI,” arguing that accuracy focus neglects computational and environmental costs\.[29](https://arxiv.org/html/2608.12875#bib.bib43)constructs Pareto frontiers of fine\-tuned BERT\-scale encoders vs\. GPT\-4o on text classification, finding encoders match or exceed frontier LLMs at 1–2 orders of magnitude lower cost\. The “overthinking” literature\([64](https://arxiv.org/html/2608.12875#bib.bib55)\)documents verbose reasoning traces and their computational cost\.[59](https://arxiv.org/html/2608.12875#bib.bib52)show that adaptive test\-time compute can outperform a uniform reasoning budget\. We study these questions on classification, similarity, clustering, and retrieval workloads\.

### 2\.2LLMs, Cross\-Document Reasoning, and Retrieval

LLMs with controllable reasoning now span closed APIs \(Gemini 3\.1 Pro, GPT\-5\([58](https://arxiv.org/html/2608.12875#bib.bib1)\), Claude Opus 4\.6\([4](https://arxiv.org/html/2608.12875#bib.bib2)\)\) and open weights \(Qwen3\([76](https://arxiv.org/html/2608.12875#bib.bib4)\), Llama 3\([30](https://arxiv.org/html/2608.12875#bib.bib3)\), and DeepSeek\-R1\([16](https://arxiv.org/html/2608.12875#bib.bib8)\), among others\)\. We evaluate ten models across six families \(§[3](https://arxiv.org/html/2608.12875#S3)\), including dense and mixture\-of\-experts models from closed and open providers\. This diversity lets us distinguish recurring patterns from effects specific to one provider\.[9](https://arxiv.org/html/2608.12875#bib.bib44)find that fine\-tuned encoder models significantly outperform zero\-shot frontier LLMs across all tested classification benchmarks, with margins ranging from 5 to over 75 F1 points depending on task granularity, consistent with our 5\.6\-point embedding advantage under the kNN vs\. zero\-shot comparison\.

Dense retrieval\([34](https://arxiv.org/html/2608.12875#bib.bib31)\)and the BEIR benchmark\([68](https://arxiv.org/html/2608.12875#bib.bib30)\)established strong zero\-shot bi\-encoder performance\. Bi\-encoders process queries and documents independently, without cross\-attention between them\. The BRIGHT benchmark\([63](https://arxiv.org/html/2608.12875#bib.bib45)\)complements BEIR with reasoning\-intensive queries\. Its top embedding model scores 18\.3 nDCG@10, while LLM\-augmented retrieval improves by up to 12\.2 points, showing the limitations of independent query and document encoding on such tasks\. RAG\([40](https://arxiv.org/html/2608.12875#bib.bib32);[25](https://arxiv.org/html/2608.12875#bib.bib33)\)pairs embedding retrieval with LLM reasoning\. We evaluate this reranking stage directly\([65](https://arxiv.org/html/2608.12875#bib.bib41)\), crossing first\-stage retrievers with cross\-encoder and LLM listwise rerankers on BEIR and BRIGHT \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\)\.[44](https://arxiv.org/html/2608.12875#bib.bib47)show CoT reranking consistently underperforms direct\-output reranking on BEIR/BRIGHT despite higher cost, the document\-ranking counterpart to our finding that reduced thinking holds or improves retrieval on all six tasks\. Appendix[F](https://arxiv.org/html/2608.12875#A6)covers LLMs as text encoders, GritLM, and the wider set of frontier models\.

## 3Experimental Setup

### 3\.1Models

#### LLMs\.

We evaluate ten frontier LLMs spanning six families: three Gemini 3 models via the Gemini API \(Gemini 3\.1 Pro, Gemini 3 Flash, Gemini 3\.1 Flash\-Lite\) and seven open\-weight models served via OpenRouter \(DeepSeek\-R1, DeepSeek\-V4\-Flash, Qwen3\.6\-27B, Qwen3\.6\-35B\-A3B, GLM\-4\.7, Kimi\-K2\.6, and MiniMax\-M2\.7\)\. Together they span dense and mixture\-of\-experts architectures, reasoning and instruct variants, and a wide capability and cost range\. Gemini 3\.1 Flash\-Lite provides a non\-reasoning baseline; the other models support extended reasoning \(chain\-of\-thought tokens\([73](https://arxiv.org/html/2608.12875#bib.bib35)\), billed at the output token rate\)\.

#### Embedding models\.

We evaluate 26 text embedding models from 118M to 14B parameters\. The multi\-model families are multilingual\-E5\([71](https://arxiv.org/html/2608.12875#bib.bib17);[72](https://arxiv.org/html/2608.12875#bib.bib18)\), Qwen3\-Embedding\([78](https://arxiv.org/html/2608.12875#bib.bib38)\), F2LLM\-v2\([79](https://arxiv.org/html/2608.12875#bib.bib76)\), Jina\-v5\([3](https://arxiv.org/html/2608.12875#bib.bib81)\), and GTE\-Qwen2\([42](https://arxiv.org/html/2608.12875#bib.bib19)\)\. We also include models from Tencent\([80](https://arxiv.org/html/2608.12875#bib.bib37)\), NVIDIA\([6](https://arxiv.org/html/2608.12875#bib.bib80)\), Salesforce\([47](https://arxiv.org/html/2608.12875#bib.bib39)\), Snowflake\([77](https://arxiv.org/html/2608.12875#bib.bib74)\), and Google\([70](https://arxiv.org/html/2608.12875#bib.bib79)\), together with E5\-Mistral\([72](https://arxiv.org/html/2608.12875#bib.bib18)\), GritLM\([48](https://arxiv.org/html/2608.12875#bib.bib20)\), Linq\-Embed\-Mistral\([15](https://arxiv.org/html/2608.12875#bib.bib75)\), BGE\-M3\([13](https://arxiv.org/html/2608.12875#bib.bib21)\), and Octen\-8B\. All embedding models run locally on a single NVIDIA H100 80GB HBM3 GPU\. Table[1](https://arxiv.org/html/2608.12875#S4.T1)lists every model with its score and cost; full details appear in Appendix[B](https://arxiv.org/html/2608.12875#A2)\.

### 3\.2Tasks and Evaluation Protocol

We introduceMTEB\(LLM\), a 37\-task benchmark covering classification, STS, clustering, pair classification, and retrieval, implemented with the MTEB framework\([49](https://arxiv.org/html/2608.12875#bib.bib10)\)\. It follows MTEB’s task and result interfaces, allowing new models and datasets to use the same evaluation pipeline\. Each MTEB\(LLM\) task is a new LLM\-specific evaluation dataset derived from the corresponding original MTEB task \(fixed seed = 42; Appendix[B\.5](https://arxiv.org/html/2608.12875#A2.SS5)\), ensuring all submissions are evaluated on identical data\.

#### Why these tasks\.

We use subsets rather than MTEB\(eng, v2\) because generative evaluation is orders of magnitude more expensive: one pass over the full suite would cost hundreds of dollars per LLM, and corpus\-in\-context retrieval requires the corpus to fit in the prompt\. Tasks are selected to span all five categories, cover multiple languages and domains, and keep retrieval corpora small enough for shorter\-context models\. MTEB’s task\-selection analysis was also designed around embedding models, so v2 is not automatically the right frame for a cross\-paradigm comparison\.

We compare complete deployment pipelines: embedding models use kNN for classification, cosine similarity for STS and retrieval, andkk\-means for clustering; LLMs receive zero\-shot prompts\. This comparison reflects a common deployment setting in which practitioners consider LLMs because labelled data are scarce\. Our few\-shot ablation \(§[4\.7](https://arxiv.org/html/2608.12875#S4.SS7)\) tests whether a small number of in\-context examples narrows the resulting gap\.

#### Classification \(8 tasks\)\.

Sentiment \(IMDB, ToxicConversations, TweetSentiment\), intent detection \(Banking77: 77 classes; MassiveIntent: 60 intents\), and multilingual classification \(AmazonCounterfactual, MTOPDomain, MassiveScenario\)\.*LLM:*Zero\-shot structured output for each test split sample\.*Embedding:*kNN trained on embeddings on the labelled training split, then used for test split predictions\.*Metric:*Accuracy\.

#### Semantic Textual Similarity \(10 tasks\)\.

English benchmarks \(STSBenchmark, SICK\-R, STS12–16\), a biomedical benchmark \(BIOSSES\), and multilingual benchmarks \(STS17, STS22v2\)\.*LLM:*Rates similarity on each task’s native scale, returning a float\.*Embedding:*Cosine similarity between embeddings\.*Metric:*Spearman correlation\.

#### Clustering \(9 tasks\)\.

Scientific papers \(ArXiv, BioRxiv, MedRxiv\) and online discussions \(Reddit, StackExchange, TwentyNewsgroups\)\.*LLM:*All documents in a single prompt with sequential identifiers; the model receives the ground\-truth cluster countkkand returns cluster assignments as a JSON array\.*Embedding:*kk\-means on document embeddings\.*Metric:*V\-measure\.

#### Pair Classification \(4 tasks\)\.

Duplicate detection \(SprintDuplicateQuestions\), paraphrase identification \(TwitterURLCorpus\), legal matching \(LegalBenchPC\), and entailment \(RTE3, multilingual\)\.*LLM:*Binary output per pair with reasoning\.*Embedding:*Cosine similarity with threshold sweep \(MTEB convention; best threshold chosen post\-hoc on the test set\)\.*Metric:*Average precision and accuracy\.

#### Retrieval \(6 tasks\)\.

A domain\-diverse suite: legal \(AILAStatutes; LegalBench consumer\-contracts QA\([31](https://arxiv.org/html/2608.12875#bib.bib62)\)\), finance \(HC3\-Finance\([32](https://arxiv.org/html/2608.12875#bib.bib78)\)\), public\-health QA, French QA \(FQuAD\([19](https://arxiv.org/html/2608.12875#bib.bib77)\)\), and Danish social media \(TwitterHjerne\([69](https://arxiv.org/html/2608.12875#bib.bib72)\)\)\.*LLM:*Corpus\-in\-context\([38](https://arxiv.org/html/2608.12875#bib.bib24)\): the full corpus is placed in the prompt with sequential identifiers; prompt caching\([28](https://arxiv.org/html/2608.12875#bib.bib48)\)amortises the corpus prefix across queries\. Corpora are deliberately small \(82–415 documents\) so that models with shorter context windows can be evaluated on identical data; this keeps the whole corpus readable in one prompt, which production\-scale retrieval does not permit \(see §[5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px2)\)\.*Embedding:*Cosine similarity ranking\.*Metric:*Recall@1 \(nDCG@kkresults in released data files\)\.

### 3\.3Cost Methodology

#### Embedding costs\.

We measure maximum\-throughput token processing on an H100 80GB at sequence length 512 \(largest fitting batch\) and compute cost asrGPU/Tr\_\{\\text\{GPU\}\}/T, whererGPU=$2\.49r\_\{\\text\{GPU\}\}=\\$2\.49/hr \(Lambda Labs H100 spot, March 2026\) andTTis tokens per hour\. Costs range from $0\.001 \(mE5\-small\) to $0\.22 \(F2LLM\-v2\-14B\); full per\-model throughput and cost data appear in Table[18](https://arxiv.org/html/2608.12875#A8.T18)\(Appendix[H\.2](https://arxiv.org/html/2608.12875#A8.SS2)\)\.

#### LLM costs\.

Token usage is extracted from each provider’susage\_stats\. We bill every non\-input \(generated\) token at the output rate:

Cost=\(input−cached\)⋅rin\+cached⋅rcache\+\(total−input\)⋅rout\\text\{Cost\}=\(\\text\{input\}\-\\text\{cached\}\)\\cdot r\_\{\\text\{in\}\}\+\\text\{cached\}\\cdot r\_\{\\text\{cache\}\}\+\(\\text\{total\}\-\\text\{input\}\)\\cdot r\_\{\\text\{out\}\}\(1\)wherercache=rin/10r\_\{\\text\{cache\}\}=r\_\{\\text\{in\}\}/10\. Reasoning \(“thinking”\) tokens are part of the generated total and are billed at the output rate, matching provider billing; providers report them differently, and we attribute each model’s reasoning share from its usage statistics \(Appendix[H\.4](https://arxiv.org/html/2608.12875#A8.SS4)\)\. Gemini models are priced at Gemini API rates and open\-weight models at OpenRouter public rates \(March/June 2026\)\. Costs range from $3\.16 \(DeepSeek\-V4\-Flash\) to $154\.14 \(Gemini 3\.1 Pro\); the complete token\-level breakdown is in Table[17](https://arxiv.org/html/2608.12875#A8.T17)\(Appendix[H\.1](https://arxiv.org/html/2608.12875#A8.SS1)\)\.

### 3\.4Throughput Methodology

We serve both paradigms on the same GPU, giving the throughput comparison common hardware and removing API rate limits\. This differs from our cost accounting \(§[3\.3](https://arxiv.org/html/2608.12875#S3.SS3)\), which uses API rates for LLMs and GPU\-rental rates for embeddings\. We serve the two open\-weight LLMs that fit on a single H100 \(Qwen3\.6\-27B dense, Qwen3\.6\-35B\-A3B MoE\) under vLLM, and run embedding models batched on the same GPU, measuring both in tokens per second \(configuration in Appendix[H\.4](https://arxiv.org/html/2608.12875#A8.SS4)\)\. Gemini is API\-only, and larger open models such as DeepSeek\-R1 need multiple GPUs\. This end\-to\-end comparison uses each paradigm’s standard inference procedure: encoder batching for embeddings and autoregressive decoding for LLMs \(Appendix[G](https://arxiv.org/html/2608.12875#A7)\)\. Per\-model embedding throughput is reported in Table[18](https://arxiv.org/html/2608.12875#A8.T18)\(Appendix[H\.2](https://arxiv.org/html/2608.12875#A8.SS2)\); the comparison appears in Figure[7](https://arxiv.org/html/2608.12875#A4.F7)\(Appendix[D\.1](https://arxiv.org/html/2608.12875#A4.SS1)\)\.

## 4Results

### 4\.1Overall Performance

Across all 36 models, the best LLM and the best embedding model are effectively tied\. Gemini 3\.1 Pro leads with a mean score of 77\.6 on MTEB\(LLM\), followed closely by Octen\-8B \(77\.2\) and Qwen3\-E\-8B \(77\.0; full ranking in Appendix[B\.2](https://arxiv.org/html/2608.12875#A2.SS2)\)\. Gemini 3 Flash scores 75\.2, and Flash\-Lite scores 64\.5\. The reasoning\-capable Gemini models occupy the top LLM ranks\.

### 4\.2Task\-Category Analysis

The aggregate tie hides large differences between task categories\. Table[1](https://arxiv.org/html/2608.12875#S4.T1)reports per\-category scores for the top models, and full per\-category rankings for all 36 models appear in Appendix[D](https://arxiv.org/html/2608.12875#A4)\. Figure[6](https://arxiv.org/html/2608.12875#A4.F6)\(Appendix[D\.1](https://arxiv.org/html/2608.12875#A4.SS1)\) shows this at the task level: some embedding model matches the best LLM on 7 of 8 classification and 7 of 10 STS tasks, but on only 1 of 6 retrieval tasks\. Retrieval is where the paradigms separate; elsewhere the choice is about cost, not quality\.

Table 1:Model overview and per\-category results\.All ten LLMs and the ten highest\-scoring embedding models\. Scores are category means on a 0–100 scale; Overall is their mean\. Cost is one MTEB\(LLM\) pass using API rates for LLMs and H100 throughput at $2\.49/hr for embeddings \(§[3\.3](https://arxiv.org/html/2608.12875#S3.SS3)\)\. Bold = best shown; full results are in Appendix[B\.2](https://arxiv.org/html/2608.12875#A2.SS2)\.#### Retrieval \(\+\+8\.5\): LLMs lead\.

Pro leads the best embedding on retrieval \(64\.5 vs\. 56\.0\) and wins five of the six retrieval tasks, losing only legal statute retrieval \(Appendix[C\.2](https://arxiv.org/html/2608.12875#A3.SS2)\)\.

#### Classification \(−\-5\.6\): embeddings lead\.

SFR\-2\([47](https://arxiv.org/html/2608.12875#bib.bib39)\)\(90\.8\) outscores Pro \(85\.2\) by a wide margin\. The gap widens on fine\-grained tasks \(Banking77: 77 classes; MassiveIntent: 60 intents\), a consequence of the kNN\-vs\-zero\-shot setup we adopt for deployment realism \(§[3\.2](https://arxiv.org/html/2608.12875#S3.SS2)\); see §[5](https://arxiv.org/html/2608.12875#S5)for the structural argument\.

#### Clustering, STS, and pair classification: statistical ties\.

On clustering the best embedding \(SFR\-2, 66\.7\) edges Pro \(66\.6\); on STS Qwen3\-E\-4B\([78](https://arxiv.org/html/2608.12875#bib.bib38)\)\(88\.8\) edges Pro \(88\.5\); on pair classification KaLM\-12B\([80](https://arxiv.org/html/2608.12875#bib.bib37)\)\(87\.1\) leads Pro \(83\.2\)\. All three rest on geometric proximity or labelled\-reference matching, which cosine similarity over learned embeddings captures directly\.

#### Statistical significance\.

A paired bootstrap test \(10,000 resamples\) places the overall difference between Pro and Octen\-8B within statistical noise \(Δ\\Delta=\+\+0\.3,pp= 0\.85, 95% CI = \[−\-2\.4,\+\+3\.1\]\)\. Category\-level tests favour LLMs on retrieval and embeddings on classification; clustering, STS, and pair classification are statistically even \(Appendix[A](https://arxiv.org/html/2608.12875#A1), Table[2](https://arxiv.org/html/2608.12875#A1.T2)\)\.

The category\-level cost–performance frontiers show the same division \(Figure[2](https://arxiv.org/html/2608.12875#S4.F2)\): Pro extends the retrieval frontier, while embeddings define the frontiers for classification, clustering, STS, and pair classification\.

Figure 2:Cost vs\. performance by task category\.Each panel plots score \(0–100\) against cost per benchmark pass \(log scale\) for all 36 models; the line is the Pareto frontier over all models and the star marks the best LLM in that category\. Pro extends the retrieval frontier; embedding models define the frontiers for classification, clustering, STS, and pair classification\.

### 4\.3Retrieve\-then\-Rerank

The corpus\-in\-context comparison above pits single\-vector embeddings against LLMs directly; production systems often insert a middle stage: a cross\-encoder or LLM reranker over a first\-stage shortlist\. We evaluate this pipeline on the standard IR benchmarks BEIR \(semantic\) and BRIGHT \(reasoning\-heavy\), crossing four first\-stage retrievers with cross\-encoder and LLM listwise rerankers \(Figure[3](https://arxiv.org/html/2608.12875#S4.F3); the full matrix is in Appendix[D\.3](https://arxiv.org/html/2608.12875#A4.SS3), Table[16](https://arxiv.org/html/2608.12875#A4.T16)\)\. The result follows the same task\-dependent pattern: on reasoning\-heavy BRIGHT, an LLM listwise reranker improves a strong embedding first stage from 22\.3 to 35\.1 nDCG@10\. On semantic BEIR, the strong embedding alone scores 63\.1, ahead of the best reranked configuration at 60\.3\. Cost scales with shortlist size: reranking a top\-100 shortlist costs $10–30 per benchmark, against $154 for reading the full corpus in context\. The MoE Qwen3\.6\-35B\-A3B is cheaper still, reaching 33\.6 nDCG@10 on BRIGHT for $10 where Qwen3\.6\-27B reaches 35\.1 for $30\.

Figure 3:Retrieve\-then\-rerank\.LLM listwise reranking improves every first stage on BRIGHT\. On BEIR, a strong embedding first stage outperforms all reranked configurations\.
### 4\.4Cost and Throughput

#### Embeddings dominate the Pareto frontier\.

The Pareto frontier contains the embedding models and Gemini 3\.1 Pro, which costs 1,431×\\timesas much as the best embedding of comparable quality \(Figure[1](https://arxiv.org/html/2608.12875#S0.F1)\)\. Pro’s marginal 0\.4\-point aggregate edge over Octen\-8B comes with a benchmark cost of $154\.14 versus $0\.11\.

#### The thinking\-token tax\.

The structural driver of LLM cost is internal reasoning \(Equation[1](https://arxiv.org/html/2608.12875#S3.E1); Figure[4](https://arxiv.org/html/2608.12875#S4.F4)\)\. Reasoning tokens account for 28–81% of inference cost across the reasoning models\. Flash\-Lite provides a non\-reasoning reference point and ranks below most embeddings\. This spending buys little on most tasks: lower reasoning budgets preserve or improve retrieval for four of six models \(§[4\.6](https://arxiv.org/html/2608.12875#S4.SS6)\)\.

![Refer to caption](https://arxiv.org/html/2608.12875v1/thinking_tax.png)Figure 4:The thinking\-token tax: what reasoning costs, and what it buys\.*\(a\)*API cost per benchmark pass by token type for all ten LLMs\.*\(b\)*Mean retrieval score with default vs\. disabled reasoning for six models from five families; labels show the reduction in generated tokens\. Four models preserve or improve retrieval with 54–96% fewer generated tokens; the two Qwen models lose ground\.
#### Cost sensitivity\.

The 1,431×\\timesratio assumes H100 spot pricing \($2\.49/hr\)\. Under alternative hardware and pricing the ratio ranges from 338×\\times\(commercial embedding API at $0\.10/MTok\) to 2,424×\\times\(L4 GPU at $0\.49/hr\)\. The order\-of\-magnitude gap persists across all evaluated cost assumptions \(Appendix[H\.3](https://arxiv.org/html/2608.12875#A8.SS3)\)\.

### 4\.5Throughput Constraints

Served on the same H100, embedding models process tokens far faster than LLMs \(Figure[7](https://arxiv.org/html/2608.12875#A4.F7), Appendix[D\.1](https://arxiv.org/html/2608.12875#A4.SS1)\)\. The two open\-weight LLMs that fit on a single H100 \(Qwen3\.6\-27B, Qwen3\.6\-35B\-A3B\) sustain 5,400–5,900 tokens/second, whereas embedding models range from 14,700 tok/s \(the largest, F2LLM\-14B\) to 4\.3M tok/s \(the smallest, mE5\-small\), a 2\.5×\\timesto 736×\\timesadvantage on identical hardware\. With hardware and serving stack fixed, the measured gap reflects the different inference procedures: autoregressive decoding and a single encoder pass\. At production scale \(millions of documents and continuous ingestion\), this throughput gap limits the practicality of LLM\-based pipelines\.

### 4\.6Ablation: Reduced Thinking

We re\-evaluate Gemini 3 Flash withreasoning\_effort=lowand, for the open models, with reasoning disabled at the serving layer, cutting reasoning tokens by 54–96% on the retrieval tasks \(Table[14](https://arxiv.org/html/2608.12875#A3.T14); cross\-family results in Figure[4](https://arxiv.org/html/2608.12875#S4.F4)b; details in Appendix[C](https://arxiv.org/html/2608.12875#A3)\)\. Two findings stand out\. First, reduced reasoning preserves or improves all six Flash retrieval scores and four of six cross\-family averages\. Second, classification changes by less than one point on all tested tasks \(\|Δ\|<1\.0\|\\Delta\|<1\.0\)\. The retrieval advantage largely persists under reduced reasoning budgets, indicating a limited role for additional test\-time reasoning\.

### 4\.7Ablation: Few\-Shot Classification

To test whether labelled examples can close the classification gap, we evaluate Flash with 5 in\-context examples per task \(Table[15](https://arxiv.org/html/2608.12875#A3.T15); details in Appendix[C](https://arxiv.org/html/2608.12875#A3)\)\. Five\-shot results are similar to or worse than the zero\-shot scores on the small\-label tasks\. Note that the embedding kNN classifier uses the full labelled train set\. The LLM prompt only gets five examples here, yet given these results, it seems unlikely more would help\.

## 5Discussion

In aggregate, the two paradigms tie on maximum performance\. The LLM advantage is narrow and specific: reasoning\-heavy retrieval, where bi\-encoders score systematically low\([63](https://arxiv.org/html/2608.12875#bib.bib45)\)\. Pro leads on 5/6 retrieval datasets, showing a consistent advantage across the suite\. Capability profiles show this split: LLMs lead on retrieval while embeddings provide balanced, often superior, coverage across the other four categories \(Appendix[D\.1](https://arxiv.org/html/2608.12875#A4.SS1), Figure[8](https://arxiv.org/html/2608.12875#A4.F8)\)\.

Viewed through the lens of information retrieval, our results reproduce the classic bi\-encoder versus cross\-encoder tradeoff at LLM scale\([55](https://arxiv.org/html/2608.12875#bib.bib16);[50](https://arxiv.org/html/2608.12875#bib.bib25)\)\. One design choice separates the four architectures we evaluate \(Figure[5](https://arxiv.org/html/2608.12875#S5.F5)\): how many documents the model is allowed to read jointly with the query\. Embedding pipelines encode queries and documents independently: each document is processed once, offline, and each query requires only vector comparisons\. A cross\-encoder reranker reads one document at a time, an LLM listwise reranker reads the top\-kkshortlist at once, and an LLM in corpus\-in\-context mode is the extreme case, reading the entire corpus in a single forward pass\. Cost follows that ordering because query\-specific computation repeats for each query: reranking scales with the shortlistkk, and corpus\-in\-context processing scales with the corpusNN\. Quality follows the same ordering, which is why reranking is a more economical way to add reasoning than placing the full corpus in context \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2608.12875v1/architecture_schematic.png)Figure 5:How many documents each architecture reads jointly with the query\.*Top:*the pipeline\.*Bottom:*its attention mask, drawn over the sameNNdocuments at the same scale in every panel so the four are directly comparable\. Each row is a token and each column a token it may read\. The red region is what one forward pass reads jointly with the query; it cannot be computed before the query arrives, so cost grows with it\.Our retrieve\-then\-rerank experiment \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\) shows the same pattern in a production\-style setting: a reranker helps most on reasoning\-heavy retrieval and little on semantic retrieval\. The same\-hardware results attribute much of the quality–cost gap to the underlying inference procedures\.

The classification pipelines receive different supervision\. Embedding classifiers use kNN over the full labelled training distribution; the LLM receives label names, task descriptions, and up to five examples\. Fine\-grained tasks make this difference especially visible \(§[4\.7](https://arxiv.org/html/2608.12875#S4.SS7)\)\. Leading embedding models are also contrastively trained on data overlapping these domains and label spaces, so part of the gap reflects in\-domain fit rather than a general representational advantage\. The comparison is therefore between deployment pipelines as practitioners meet them, not intrinsic ceilings: a classification\-post\-trained LLM would likely narrow the gap, and none was available among the frontier models we evaluate\.

The thinking\-token tax reveals a mismatch between default reasoning budgets and these tasks\. Reasoning improves the overall Flash score, while lower reasoning budgets preserve retrieval quality for most evaluated models\. Where direct reading comprehension suffices, the model reconsiders straightforward relevance judgements, making the default reasoning budget wasteful for most of these benchmark tasks\([44](https://arxiv.org/html/2608.12875#bib.bib47)\)\. For practitioners, reducing reasoning effort preserves retrieval quality for most evaluated models while removing the largest component of LLM cost\.

#### Deployment implications\.

These findings motivate a hybrid architecture: embedding models provide high\-throughput candidate retrieval, followed by LLM reasoning over a shortlist \(Figure[7](https://arxiv.org/html/2608.12875#A4.F7)\)\. This division of labour supports retrieve\-then\-reason pipelines\([40](https://arxiv.org/html/2608.12875#bib.bib32);[25](https://arxiv.org/html/2608.12875#bib.bib33)\); our reranker results \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\) show where they help\. Across the four non\-retrieval categories, small\-to\-medium embedding models closely match the best LLM at a fraction of the cost\. Hardware improvements will reduce the absolute cost of both paradigms\. Their different inference procedures preserve an embedding throughput advantage for similarity, classification, and clustering workloads \(§[3\.4](https://arxiv.org/html/2608.12875#S3.SS4)\)\. Accuracy\-only leaderboards obscure large cost differences between similarly scored systems\. We recommend reporting Pareto frontiers and significance tests alongside accuracy\([10](https://arxiv.org/html/2608.12875#bib.bib49);[17](https://arxiv.org/html/2608.12875#bib.bib50)\)\.

#### Limitations\.

Our ten LLMs from six families are a snapshot of a fast\-moving frontier; because MTEB\(LLM\) uses the MTEB framework, future models can be evaluated with the same pipeline\. The corpus\-in\-context protocol places corpora of 82–415 documents entirely in the prompt, which is only feasible at small scale: production corpora require an indexed first stage, so the cost gap we report is a lower bound\. We complement this setting with a bi\-encoder, cross\-encoder reranker, and LLM listwise reranker comparison on BEIR and BRIGHT \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\)\. Embeddings also benefit from post\-hoc threshold optimisation in pair classification \(MTEB convention\) with no LLM analogue\. Appendix[G](https://arxiv.org/html/2608.12875#A7)gives the full set\.

## 6Conclusion

We present MTEB\(LLM\), the first cost\-aware comparison of LLM and embedding deployment pipelines across five MTEB task categories, evaluating ten LLMs from six families and 26 embedding models on 37 tasks\. The two paradigms match in aggregate but differ by task: LLMs lead on reasoning\-heavy retrieval, embeddings lead on classification, and the rest are tied\. Comparable LLM quality comes with substantially higher cost and lower throughput\.

Embedding pipelines therefore remain the cost\-efficient default, with reasoning\-intensive retrieval the clearest case for an LLM\. Our retrieve\-then\-rerank results support a hybrid strategy: embeddings for candidate retrieval, LLMs for reasoning over the shortlist\.

Pareto frontiers add deployment cost and throughput to accuracy\-based rankings, and we encourage their wider use\. MTEB\(LLM\) is released with MTEB\-compatible code and datasets, so the evaluation can expand as new models appear\.

## Acknowledgments

We are extremely thankful to Laude Institute for supporting this work\.

## Reproducibility Statement

Code, raw results, and analysis scripts are released at[https://github\.com/embeddings\-benchmark/embedders\-dilemma](https://github.com/embeddings-benchmark/embedders-dilemma)in an implementation built on the MTEB framework\. The LLM\-specific datasets are hosted on Hugging Face undermteb/llm\-eval\-\*, and each task pins its dataset to an exact revision\. LLM prompt templates, schema validation logic, and token\-usage extraction scripts are included in the release, and every figure and table in the paper regenerates from the released result files\. The provided scripts reproduce the embedding throughput benchmarks on an NVIDIA H100 GPU\. Detailed experimental configurations are described in §[3](https://arxiv.org/html/2608.12875#S3)and Appendix[I](https://arxiv.org/html/2608.12875#A9)\.

## Ethics Statement

This work compares publicly available models on standard benchmarks and does not involve human subjects or private data\. All evaluated datasets are publicly hosted and have been previously released for research purposes\. Cost and throughput reporting may influence deployment decisions; we encourage practitioners to consider the environmental impact of LLM\-based pipelines, including the carbon footprint of chain\-of\-thought inference, alongside the economic factors discussed in this paper\. Reducing reasoning budgets can preserve performance on some tasks while lowering compute use\. LLM assistance was used during the preparation of this paper for writing polish, phrasing refinement, and code formatting \(e\.g\., matplotlib styling\); no LLM was used to originate research ideas, generate evaluation data, produce plots, or evaluate model outputs\. All scientific content, experimental design, and conclusions are the authors’ own\.

## References

- Agirreet al\.\(2016\)E\. Agirre, C\. Banea, D\. Cer, M\. Diab, A\. Gonzalez\-Agirre, R\. Mihalcea, G\. Rigau, and J\. WiebeSemEval\-2016 task 1: semantic textual similarity, monolingual and cross\-lingual evaluation\.InProceedings of the 10th International Workshop on Semantic Evaluation \(SemEval\-2016\),S\. Bethard, M\. Carpuat, D\. Cer, D\. Jurgens, P\. Nakov, and T\. Zesch \(Eds\.\),San Diego, California,pp\. 497–511\.External Links:[Link](https://aclanthology.org/S16-1081/),[Document](https://dx.doi.org/10.18653/v1/S16-1081)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.17.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.18.6)\.
- Agirreet al\.\(2012\)E\. Agirre, D\. Cer, M\. Diab, and A\. Gonzalez\-AgirreSemEval\-2012 task 6: a pilot on semantic textual similarity\.In\*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation \(SemEval 2012\),E\. Agirre, J\. Bos, M\. Diab, S\. Manandhar, Y\. Marton, and D\. Yuret \(Eds\.\),Montréal, Canada,pp\. 385–393\.External Links:[Link](https://aclanthology.org/S12-1051/)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.14.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.15.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.16.6)\.
- Akramet al\.\(2026\)M\. K\. Akram, S\. Sturua, N\. Havriushenko, Q\. Herreros, M\. Günther, M\. Werk, and H\. XiaoJina\-embeddings\-v5\-text: task\-targeted embedding distillation\.External Links:2602\.15547,[Link](https://arxiv.org/abs/2602.15547)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.19.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.21.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Anthropic \(2026\)AnthropicClaude Opus 4\.6\.Model CardAnthropic\.External Links:[Link](https://www.anthropic.com/claude/opus)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1),[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p2.1),[Appendix G](https://arxiv.org/html/2608.12875#A7.SS0.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Assadiet al\.\(2025\)A\. E\. Assadi, I\. Chung, R\. Solomatin, N\. Muennighoff, and K\. EnevoldsenHUME: measuring the human\-model performance gap in text embedding tasks\.External Links:2510\.10062,[Link](https://arxiv.org/abs/2510.10062)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p4.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Babakhinet al\.\(2025\)Y\. Babakhin, R\. Osmulski, R\. Ak, G\. Moreira, M\. Xu, B\. Schifferer, B\. Liu, and E\. OldridgeLlama\-embed\-nemotron\-8b: a universal text embedding model for multilingual and cross\-lingual tasks\.External Links:2511\.07025,[Link](https://arxiv.org/abs/2511.07025)Cited by:[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- BehnamGhaderet al\.\(2024\)P\. BehnamGhader, V\. Adlakha, M\. Mosbach, D\. Bahdanau, N\. Chapados, and S\. ReddyLLM2Vec: large language models are secretly powerful text encoders\.External Links:2404\.05961,[Link](https://arxiv.org/abs/2404.05961)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p2.1)\.
- Borkanet al\.\(2019\)D\. Borkan, L\. Dixon, J\. Sorensen, N\. Thain, and L\. VassermanNuanced metrics for measuring unintended bias with real data for text classification\.External Links:1903\.04561,[Link](https://arxiv.org/abs/1903.04561)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.9.6)\.
- Bucher and Martini \(2024\)M\. J\. J\. Bucher and M\. MartiniFine\-tuned ‘small’ LLMs \(still\) significantly outperform zero\-shot generative AI models in text classification\.External Links:2406\.08660,[Link](https://arxiv.org/abs/2406.08660)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Cardet al\.\(2020\)D\. Card, P\. Henderson, U\. Khandelwal, R\. Jia, K\. Mahowald, and D\. JurafskyWith little power comes great responsibility\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 9263–9274\.External Links:[Link](https://arxiv.org/abs/2010.06595)Cited by:[§5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px1.p1.1)\.
- Casanuevaet al\.\(2020\)I\. Casanueva, T\. Temčinas, D\. Gerz, M\. Henderson, and I\. VulićEfficient intent detection with dual sentence encoders\.InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI,External Links:[Link](https://arxiv.org/abs/2003.04807)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.4.6)\.
- Ceret al\.\(2017\)D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. SpeciaSemEval\-2017 task 1: semantic textual similarity multilingual and cross\-lingual focused evaluation\.InProceedings of the 11th International Workshop on Semantic Evaluation \(SemEval\-2017\),External Links:[Link](https://arxiv.org/abs/1708.00055)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.12.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.20.6)\.
- Chenet al\.\(2025\)J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. LiuM3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.External Links:2402\.03216,[Link](https://arxiv.org/abs/2402.03216)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.33.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2022\)X\. Chen, A\. Zeynali, C\. Camargo, F\. Flöck, D\. Gaffney, P\. Grabowicz, S\. A\. Hale, D\. Jurgens, and M\. SamorySemEval\-2022 task 8: multilingual news article similarity\.InProceedings of the 16th International Workshop on Semantic Evaluation \(SemEval\-2022\),G\. Emerson, N\. Schluter, G\. Stanovsky, R\. Kumar, A\. Palmer, N\. Schneider, S\. Singh, and S\. Ratan \(Eds\.\),Seattle, United States,pp\. 1094–1106\.External Links:[Link](https://aclanthology.org/2022.semeval-1.155/),[Document](https://dx.doi.org/10.18653/v1/2022.semeval-1.155)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.21.6)\.
- Choiet al\.\(2024\)C\. Choi, J\. Kim, S\. Lee, J\. Kwon, S\. Gu, Y\. Kim, M\. Cho, and J\. SohnLinq\-embed\-mistral technical report\.External Links:2412\.03223,[Link](https://arxiv.org/abs/2412.03223)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.24.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang,et al\.DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Dehghaniet al\.\(2021\)M\. Dehghani, Y\. Tay, A\. A\. Gritsenko, Z\. Zhao, N\. Houlsby, F\. Diaz, D\. Metzler, and O\. VinyalsThe benchmark lottery\.External Links:2107\.07002,[Link](https://arxiv.org/abs/2107.07002)Cited by:[§5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px1.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 4171–4186\.External Links:[Link](https://arxiv.org/abs/1810.04805)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- d’Hoffschmidtet al\.\(2020\)M\. d’Hoffschmidt, W\. Belblidia, Q\. Heinrich, T\. Brendlé, and M\. VidalFQuAD: french question answering dataset\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1193–1208\.External Links:[Link](https://arxiv.org/abs/2002.06071)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.39.6),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Efron \(1979\)B\. EfronBootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:ISSN 00905364, 21688966,[Link](http://www.jstor.org/stable/2958830)Cited by:[Appendix A](https://arxiv.org/html/2608.12875#A1.SS0.SSS0.Px1.p1.1)\.
- El Assadiet al\.\(2026a\)A\. El Assadi, I\. Chung, C\. Xiao, R\. Solomatin, A\. Jha, R\. Chand, S\. Singh, K\. Wang, A\. S\. Khan, M\. M\. Nasser, S\. Fong, P\. He, A\. Xiao, A\. S\. Munot, A\. Shrivastava, A\. Gazizov, N\. Muennighoff, and K\. EnevoldsenMAEB: massive audio embedding benchmark\.External Links:2602\.16008,[Link](https://arxiv.org/abs/2602.16008)Cited by:[§1](https://arxiv.org/html/2608.12875#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- El Assadiet al\.\(2026b\)A\. El Assadi, R\. Solomatin, I\. Chung, C\. Xiao, D\. Shah, M\. Dey, S\. Sudhakar, Z\. Bugaud, W\. Siblini, A\. S\. Munot, Y\. Devavarapu, R\. Ireddi, M\. Yang, M\. Kardos, N\. Muennighoff, and K\. EnevoldsenMVEB: massive video embedding benchmark\.External Links:2606\.14958,[Link](https://arxiv.org/abs/2606.14958)Cited by:[§1](https://arxiv.org/html/2608.12875#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Enevoldsenet al\.\(2025\)K\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzeminski, G\. I\. Winata, S\. Sturua, S\. Utpala, M\. Ciancone, M\. Schaeffer, G\. Sequeira, D\. Misra, S\. Dhakal, J\. Rystrøm, R\. Solomatin, Ö\. Çağatan, A\. Kundu, M\. Bernstorff, S\. Xiao, A\. Sukhlecha, B\. Pahwa, R\. Poświąta, K\. K\. GV, S\. Ashraf, D\. Auras, B\. Plüster, J\. P\. Harries, L\. Magne, I\. Mohr, M\. Hendriksen, D\. Zhu, H\. Gisserot\-Boukhlef, T\. Aarsen, J\. Kostkan, K\. Wojtasik, T\. Lee, M\. Šuppa, C\. Zhang, R\. Rocca, M\. Hamdy, A\. Michail, J\. Yang, M\. Faysse, A\. Vatolin, N\. Thakur, M\. Dey, D\. Vasani, P\. Chitale, S\. Tedeschi, N\. Tai, A\. Snegirev, M\. Günther, M\. Xia, W\. Shi, X\. H\. Lù, J\. Clive, G\. Krishnakumar, A\. Maksimova, S\. Wehrli, M\. Tikhonova, H\. Panchal, A\. Abramov, M\. Ostendorff, Z\. Liu, S\. Clematide, L\. J\. Miranda, A\. Fenogenova, G\. Song, R\. B\. Safi, W\. Li, A\. Borghini, F\. Cassano, H\. Su, J\. Lin, H\. Yen, L\. Hansen, S\. Hooker, C\. Xiao, V\. Adlakha, O\. Weller, S\. Reddy, and N\. MuennighoffMMTEB: massive multilingual text embedding benchmark\.External Links:2502\.13595,[Link](https://arxiv.org/abs/2502.13595)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.38.6),[Appendix G](https://arxiv.org/html/2608.12875#A7.SS0.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2608.12875#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- FitzGeraldet al\.\(2022\)J\. FitzGerald, C\. Hench, C\. Peris, S\. Mackie, K\. Rottmann, A\. Sanchez, A\. Nash, L\. Urbach, V\. Kakarala, R\. Singh, S\. Ranganath, L\. Crist, M\. Britan, W\. Leeuwis, G\. Tur, and P\. NatarajanMASSIVE: a 1M\-example multilingual natural language understanding dataset with 51 typologically\-diverse languages\.External Links:2204\.08582,[Link](https://arxiv.org/abs/2204.08582)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.7.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.8.6)\.
- Gaoet al\.\(2024\)Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. WangRetrieval\-augmented generation for large language models: a survey\.External Links:2312\.10997,[Link](https://arxiv.org/abs/2312.10997)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1),[§5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px1.p1.1)\.
- Gemma Teamet al\.\(2025\)Gemma Teamet al\.Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1)\.
- Giampiccoloet al\.\(2007\)D\. Giampiccolo, B\. Magnini, I\. Dagan, and B\. DolanThe third PASCAL recognizing textual entailment challenge\.InProceedings of the ACL\-PASCAL Workshop on Textual Entailment and Paraphrasing,S\. Sekine, K\. Inui, I\. Dagan, B\. Dolan, D\. Giampiccolo, and B\. Magnini \(Eds\.\),Prague,pp\. 1–9\.External Links:[Link](https://aclanthology.org/W07-1401/)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.36.6)\.
- Gimet al\.\(2024\)I\. Gim, G\. Chen, S\. Lee, N\. Sarda, A\. Khandelwal, and L\. ZhongPrompt cache: modular attention reuse for low\-latency inference\.InProceedings of Machine Learning and Systems,Vol\.6\.External Links:2311\.04934,[Link](https://arxiv.org/abs/2311.04934)Cited by:[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Gonzalez \(2026\)A\. A\. V\. GonzalezCost\-aware model selection for text classification: multi\-objective trade\-offs between fine\-tuned encoders and LLM prompting in production\.External Links:2602\.06370,[Link](https://arxiv.org/abs/2602.06370)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p2.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Guhaet al\.\(2023\)N\. Guha, J\. Nyarko, D\. E\. Ho, C\. Ré, A\. Chilton, A\. Narayana, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. N\. Rockmore, D\. Zambrano, D\. Talisman, E\. Hoque, F\. Surani, F\. Fagan, G\. Correction, G\. Bassett, H\. Porat, J\. Livermore, J\. Noss, J\. Noss, K\. Ashley, K\. D\. Ashley, L\. Lan, M\. Tyler, N\. Guha, N\. Patel, R\. Thalken, T\. F\. Gordon, T\. Skorepa, W\. Xu, and Z\. ZhouLegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2308.11462)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.35.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.41.6),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Guoet al\.\(2023\)B\. Guo, X\. Zhang, Z\. Wang, M\. Jiang, J\. Nie, Y\. Ding, J\. Yue, and Y\. WuHow close is ChatGPT to human experts? comparison corpus, evaluation, and detection\.arXiv preprint arXiv:2301\.07597\.External Links:[Link](https://arxiv.org/abs/2301.07597)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.40.6),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Jianget al\.\(2023\)T\. Jiang, S\. Huang, Z\. Luan, D\. Wang, and F\. ZhuangScaling sentence embeddings with large language models\.External Links:2307\.16645,[Link](https://arxiv.org/abs/2307.16645)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p2.1)\.
- Karpukhinet al\.\(2020\)V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. YihDense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 6769–6781\.External Links:[Link](https://arxiv.org/abs/2004.04906)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1)\.
- Khattab and Zaharia \(2020\)O\. Khattab and M\. ZahariaColBERT: efficient and effective passage search via contextualized late interaction over BERT\.InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 39–48\.External Links:[Link](https://arxiv.org/abs/2004.12832)Cited by:[§F\.3](https://arxiv.org/html/2608.12875#A6.SS3.p1.1)\.
- Lanet al\.\(2017\)W\. Lan, S\. Qiu, H\. He, and W\. XuA continuously growing dataset of sentential paraphrases\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,M\. Palmer, R\. Hwa, and S\. Riedel \(Eds\.\),Copenhagen, Denmark,pp\. 1224–1234\.External Links:[Link](https://aclanthology.org/D17-1126/),[Document](https://dx.doi.org/10.18653/v1/D17-1126)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.34.6)\.
- Leeet al\.\(2025a\)C\. Lee, R\. Roy, M\. Xu, J\. Raiman, M\. Shoeybi, B\. Catanzaro, and W\. PingNV\-Embed: improved techniques for training LLMs as generalist embedding models\.InThe Thirteenth International Conference on Learning Representations,External Links:2405\.17428,[Link](https://arxiv.org/abs/2405.17428)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.17.5),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Leeet al\.\(2024\)J\. Lee, A\. Chen, Z\. Dai, D\. Dua, D\. S\. Sachan, M\. Boratko, Y\. Luan, S\. M\. R\. Arnold, V\. Perot, S\. Dalmia, H\. Hu, X\. Lin, P\. Pasupat, A\. Amini, J\. R\. Cole, S\. Riedel, I\. Naim, M\. Chang, and K\. GuuCan long\-context language models subsume retrieval, RAG, SQL, and more?\.External Links:2406\.13121,[Link](https://arxiv.org/abs/2406.13121)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Leeet al\.\(2025b\)J\. Lee, F\. Chen, S\. Dua, D\. Cer, M\. Shanbhogue, I\. Naim, G\. H\. Ábrego, Z\. Li, K\. Chen, H\. S\. Vera, X\. Ren, S\. Zhang, D\. Salz, M\. Boratko, J\. Han, B\. Chen, S\. Huang, V\. Rao, P\. Suganthan, F\. Han, A\. Doumanoglou, N\. Gupta, F\. Moiseev, C\. Yip, A\. Jain, S\. Baumgartner, S\. Shahi, F\. P\. Gomez, S\. Mariserla, M\. Choi, P\. Shah, S\. Goenka, K\. Chen, Y\. Xia, K\. Chen, S\. M\. K\. Duddu, Y\. Chen, T\. Walker, W\. Zhou, R\. Ghiya, Z\. Gleicher, K\. Gill, Z\. Dong, M\. Seyedhosseini, Y\. Sung, R\. Hoffmann, and T\. DuerigGemini embedding: generalizable embeddings from Gemini\.External Links:2503\.07891,[Link](https://arxiv.org/abs/2503.07891)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.26.5),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1),[§5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2021\)H\. Li, A\. Arora, S\. Chen, A\. Gupta, S\. Gupta, and Y\. MehdadMTOP: a comprehensive multilingual task\-oriented semantic parsing benchmark\.External Links:2008\.09335,[Link](https://arxiv.org/abs/2008.09335)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.6.6)\.
- Liet al\.\(2023\)Z\. Li, X\. Zhang, Y\. Zhang, D\. Long, P\. Xie, and M\. ZhangTowards general text embeddings with multi\-stage contrastive learning\.External Links:2308\.03281,[Link](https://arxiv.org/abs/2308.03281)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.23.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.30.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)Z\. Li, C\. Li, M\. Zhang, Q\. Mei, and M\. BenderskyRetrieval augmented generation or long\-context LLMs? a comprehensive study and hybrid approach\.External Links:2407\.16833,[Link](https://arxiv.org/abs/2407.16833)Cited by:[§F\.3](https://arxiv.org/html/2608.12875#A6.SS3.p2.1)\.
- Luet al\.\(2025\)X\. Luet al\.Rethinking reasoning in document ranking: why chain\-of\-thought falls short\.External Links:2510\.08985,[Link](https://arxiv.org/abs/2510.08985)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1),[§5](https://arxiv.org/html/2608.12875#S5.p5.1)\.
- Maaset al\.\(2011\)A\. L\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. PottsLearning word vectors for sentiment analysis\.InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,D\. Lin, Y\. Matsumoto, and R\. Mihalcea \(Eds\.\),Portland, Oregon, USA,pp\. 142–150\.External Links:[Link](https://aclanthology.org/P11-1015/)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.3.6)\.
- Marelliet al\.\(2014\)M\. Marelli, S\. Menini, M\. Baroni, L\. Bentivogli, R\. Bernardi, and R\. ZamparelliA SICK cure for the evaluation of compositional distributional semantic models\.InProceedings of the Ninth International Conference on Language Resources and Evaluation \(LREC’14\),N\. Calzolari, K\. Choukri, T\. Declerck, H\. Loftsson, B\. Maegaard, J\. Mariani, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Reykjavik, Iceland,pp\. 216–223\.External Links:[Link](https://aclanthology.org/L14-1314/)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.13.6)\.
- Menget al\.\(2024\)R\. Meng, Y\. Liu, S\. R\. Joty, C\. Xiong, Y\. Zhou, and S\. YavuzSFR\-embedding\-2: advanced text embedding with multi\-stage training\.External Links:[Link](https://huggingface.co/Salesforce/SFR-Embedding-2_R)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.20.5),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.12875#S4.SS2.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, H\. Su, L\. Wang, N\. Yang, F\. Wei, T\. Yu, A\. Singh, and D\. KielaGenerative representational instruction tuning\.External Links:2402\.09906,[Link](https://arxiv.org/abs/2402.09906)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.39.5),[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p2.1),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, N\. Tazi, L\. Magne, and N\. ReimersMTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.External Links:2210\.07316,[Link](https://arxiv.org/abs/2210.07316)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.10.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.23.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.24.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.25.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.26.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.27.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.28.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.29.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.30.6),[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.31.6),[§1](https://arxiv.org/html/2608.12875#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.p1.1)\.
- Muennighoff \(2022\)N\. MuennighoffSGPT: GPT sentence embeddings for semantic search\.External Links:2202\.08904,[Link](https://arxiv.org/abs/2202.08904)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p2.1),[§5](https://arxiv.org/html/2608.12875#S5.p2.1)\.
- Nieet al\.\(2025\)Z\. Nie, Z\. Feng, M\. Li, C\. Zhang, Y\. Zhang, D\. Long, and R\. ZhangWhen text embedding meets large language model: a comprehensive survey\.External Links:2412\.09165,[Link](https://arxiv.org/abs/2412.09165)Cited by:[footnote 1](https://arxiv.org/html/2608.12875#footnote1)\.
- OpenAI \(2023\)OpenAIGPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1)\.
- OpenAI \(2024\)OpenAIOpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1)\.
- O’Neillet al\.\(2021\)J\. O’Neill, P\. Rozenshtein, R\. Kiryo, M\. Kubota, and D\. BollegalaI wish i would have loved this one, but i didn’t – a multilingual dataset for counterfactual detection in product reviews\.External Links:2104\.06893,[Link](https://arxiv.org/abs/2104.06893)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.5.6)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,pp\. 3982–3992\.External Links:[Link](https://arxiv.org/abs/1908.10084)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.12875#S5.p2.1)\.
- Schwartzet al\.\(2020\)R\. Schwartz, J\. Dodge, N\. A\. Smith, and O\. EtzioniGreen ai\.Communications of the ACM63\(12\),pp\. 54–63\.External Links:1907\.10597,[Link](https://arxiv.org/abs/1907.10597)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p2.1)\.
- Shahet al\.\(2018\)D\. Shah, T\. Lei, A\. Moschitti, S\. Romeo, and P\. NakovAdversarial domain adaptation for duplicate question detection\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 1056–1063\.External Links:[Link](https://aclanthology.org/D18-1131/),[Document](https://dx.doi.org/10.18653/v1/D18-1131)Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.33.6)\.
- Singhet al\.\(2025\)A\. Singhet al\.OpenAI GPT\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1),[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p2.1),[Appendix G](https://arxiv.org/html/2608.12875#A7.SS0.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Snellet al\.\(2024\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling model parameters\.External Links:2408\.03314,[Link](https://arxiv.org/abs/2408.03314)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p2.1)\.
- Soğancıoğluet al\.\(2017\)G\. Soğancıoğlu, H\. Öztürk, and A\. ÖzgürBIOSSES: a semantic sentence similarity estimation system for the biomedical domain\.Bioinformatics33\(14\),pp\. i49–i58\.Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.19.6)\.
- Springeret al\.\(2025\)J\. M\. Springer, S\. Kotha, D\. Fried, G\. Neubig, and A\. RaghunathanRepetition improves language model embeddings\.InThe Thirteenth International Conference on Learning Representations,External Links:2402\.15449,[Link](https://arxiv.org/abs/2402.15449)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p2.1)\.
- Suet al\.\(2023\)H\. Su, W\. Shi, J\. Kasai, Y\. Wang, Y\. Hu, M\. Ostendorf, W\. Yih, N\. A\. Smith, L\. Zettlemoyer, and T\. YuOne embedder, any task: instruction\-finetuned text embeddings\.InFindings of the Association for Computational Linguistics: ACL 2023,External Links:2212\.09741,[Link](https://arxiv.org/abs/2212.09741)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Suet al\.\(2024\)H\. Su, H\. Yen, M\. Xia, W\. Shi, N\. Muennighoff, H\. Wang, H\. Liu, Q\. Shi, Z\. S\. Siegel, M\. Tang, R\. Sun, J\. Yoon, S\. Arik, D\. Chen, and T\. YuBRIGHT: a realistic and challenging benchmark for reasoning\-intensive retrieval\.External Links:2407\.12883,[Link](https://arxiv.org/abs/2407.12883)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1),[§5](https://arxiv.org/html/2608.12875#S5.p1.1)\.
- Suiet al\.\(2025\)Y\. Sui, Y\. Chuang, G\. Wang, J\. Zhang, T\. Zhang, J\. Yuan, H\. Liu, A\. Wen, S\. Zhong, N\. Zou, H\. Chen, and X\. HuStop overthinking: a survey on efficient reasoning for large language models\.External Links:2503\.16419,[Link](https://arxiv.org/abs/2503.16419)Cited by:[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p2.1)\.
- Sunet al\.\(2023a\)W\. Sun, L\. Yan, X\. Ma, P\. Ren, D\. Yin, and Z\. RenIs ChatGPT good at search? investigating large language models as re\-ranking agents\.External Links:2304\.09542,[Link](https://arxiv.org/abs/2304.09542)Cited by:[§F\.3](https://arxiv.org/html/2608.12875#A6.SS3.p1.1),[Appendix G](https://arxiv.org/html/2608.12875#A7.SS0.SSS0.Px3.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1)\.
- Sunet al\.\(2023b\)X\. Sun, X\. Li, J\. Li, F\. Wu, S\. Guo, T\. Zhang, and G\. WangText classification via large language models\.External Links:2305\.08377,[Link](https://arxiv.org/abs/2305.08377)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p1.1)\.
- Taoet al\.\(2024\)C\. Tao, T\. Shen, S\. Gao, J\. Zhang, Z\. Li, K\. Hua, W\. Hu, Z\. Tao, and S\. MaLlms are also effective embedding models: an in\-depth overview\.ACM Transactions on Information Systems\.Cited by:[footnote 1](https://arxiv.org/html/2608.12875#footnote1)\.
- Thakuret al\.\(2021\)N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. GurevychBEIR: a heterogenous benchmark for zero\-shot evaluation of information retrieval models\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2104.08663)Cited by:[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p2.1)\.
- Vejlgaard Holmet al\.\(2025\)S\. Vejlgaard Holm, L\. K\. Hansen, and M\. C\. NielsenR\. Johansson and S\. Stymne \(Eds\.\)Danoliteracy of generative large language models\.University of Tartu Library,Tallinn, Estonia\.External Links:[Link](https://aclanthology.org/2025.nodalida-1.78/),ISBN 978\-9908\-53\-109\-0Cited by:[Table 12](https://arxiv.org/html/2608.12875#A2.T12.2.1.43.6),[§3\.2](https://arxiv.org/html/2608.12875#S3.SS2.SSS0.Px6.p1.1)\.
- Veraet al\.\(2025\)H\. S\. Vera, S\. Dua, B\. Zhang, D\. Salz, R\. Mullins,et al\.EmbeddingGemma: powerful and lightweight text representations\.External Links:2509\.20354,[Link](https://arxiv.org/abs/2509.20354)Cited by:[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Wanget al\.\(2022\)L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. WeiText embeddings by weakly\-supervised contrastive pre\-training\.External Links:2212\.03533,[Link](https://arxiv.org/abs/2212.03533)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.34.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.36.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.37.5),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. WeiImproving text embeddings with large language models\.External Links:2401\.00368,[Link](https://arxiv.org/abs/2401.00368)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.32.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.38.5),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.External Links:[Link](https://arxiv.org/abs/2201.11903)Cited by:[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px1.p1.1)\.
- Welleret al\.\(2025\)O\. Weller, K\. Ricci, E\. Yang, A\. Yates, D\. Lawrie, and B\. V\. DurmeRank1: test\-time compute for reranking in information retrieval\.External Links:2502\.18418,[Link](https://arxiv.org/abs/2502.18418)Cited by:[§F\.3](https://arxiv.org/html/2608.12875#A6.SS3.p1.1),[Appendix G](https://arxiv.org/html/2608.12875#A7.SS0.SSS0.Px3.p1.1)\.
- Xiaoet al\.\(2025\)C\. Xiao, I\. Chung, I\. Kerboua, J\. Stirling, X\. Zhang, M\. Kardos, R\. Solomatin, N\. Al Moubayed, K\. Enevoldsen, and N\. MuennighoffMIEB: massive image embedding benchmark\.External Links:2504\.10471,[Link](https://arxiv.org/abs/2504.10471)Cited by:[§1](https://arxiv.org/html/2608.12875#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.12875#S2.SS1.p1.1)\.
- Yanget al\.\(2025\)A\. Yanget al\.Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§F\.2](https://arxiv.org/html/2608.12875#A6.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.12875#S2.SS2.p1.1)\.
- Yuet al\.\(2024\)P\. Yu, L\. Merrick, G\. Nuti, and D\. CamposArctic\-embed 2\.0: multilingual retrieval without compromise\.External Links:2412\.04506,[Link](https://arxiv.org/abs/2412.04506)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.35.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhanget al\.Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176,[Link](https://arxiv.org/abs/2506.05176)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.15.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.16.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.25.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.12875#S4.SS2.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, Z\. Liao, H\. Yu, P\. Di, and R\. WangF2LLM\-v2: inclusive, performant, and efficient embeddings for a multilingual world\.External Links:2603\.19223,[Link](https://arxiv.org/abs/2603.19223)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.22.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.27.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.28.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.29.5),[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.31.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2025\)X\. Zhao, X\. Hu, Z\. Shan, Z\. Sun, Z\. Liu, D\. Li, S\. Ye, X\. Wei, Q\. Chen, B\. Hu, H\. Wang, J\. Yu, and M\. ZhangKaLM\-Embedding\-V2: superior training techniques and data inspire a versatile embedding model\.External Links:2506\.20923,[Link](https://arxiv.org/abs/2506.20923)Cited by:[Table 3](https://arxiv.org/html/2608.12875#A2.T3.2.18.5),[§3\.1](https://arxiv.org/html/2608.12875#S3.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.12875#S4.SS2.SSS0.Px3.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§F\.1](https://arxiv.org/html/2608.12875#A6.SS1.p1.1)\.

## Appendix AStatistical Significance Testing

#### Bootstrap procedure\.

We implement a paired bootstrap test\([20](https://arxiv.org/html/2608.12875#bib.bib51)\)comparing Gemini 3\.1 Pro against the best embedding model in each comparison\. The resampling unit is the task level: each bootstrap replicate draws MTEB\(LLM\) tasks with replacement from the full MTEB\(LLM\) task set, and the score difference is computed as the mean\-of\-means difference across the resampled task set\. Task\-level resampling accommodates the different metrics used across categories \(accuracy, Spearmanρ\\rho, V\-measure, average precision, and Recall@1\)\. Equal task weighting matches the macro\-average score\.

We report unadjustedpp\-values for five planned category\-level comparisons\. Under a Bonferroni threshold ofα\\alpha= 0\.01, classification remains significant and retrieval falls above the threshold\.

Table[2](https://arxiv.org/html/2608.12875#A1.T2)reports the full results\. The overall comparison \(Pro vs\. Octen\-8B across MTEB\(LLM\)\) yieldsΔ\\Delta=\+\+0\.3, 95% CI = \[−\-2\.4, \+3\.1\],pp= 0\.85, a statistical tie\. Category\-level tests identify two significant differences:

- •Classification:embeddings win \(Δ\\Delta=−\-5\.6, CI = \[−\-9\.2,−\-2\.4\],pp<<0\.01\)
- •Retrieval:LLMs win \(Δ\\Delta= \+8\.5, CI = \[\+0\.2, \+16\.8\],pp<<0\.05\)

STS \(pp= 0\.74\), clustering \(pp= 0\.97\), and pair classification \(pp= 0\.49\) show no significant paradigm difference\.

Table 2:Statistical significance\.Paired bootstrap test \(10,000 resamples, seed 42\)\.Δ\\Delta= Gemini 3\.1 Pro−\-best embedding; significance atα\\alpha= 0\.05\. Pair classification uses Pro for consistency, although Flash scores higher\.

## Appendix BEvaluated Models and Tasks

This section lists the evaluated models, category\- and task\-level results, and benchmark tasks and metrics\.

### B\.1Model Overview

Table[3](https://arxiv.org/html/2608.12875#A2.T3)lists all 36 evaluated models — 10 LLMs and 26 embedding models — with parameter counts, mean MTEB\(LLM\) scores, total benchmark costs, and paper references\. Models range from mE5\-small \(118M parameters, $0\.001\) to Gemini 3\.1 Pro \($154\.14\)\. Models without a dedicated paper are marked “–”\.

ModelParamsScoreCostReference*LLM Models*Gemini 3\.1 Pro–77\.6$154\.14–Gemini 3 Flash–75\.2$55\.87–Qwen3\.6\-27B–73\.9$103\.43–Qwen3\.6\-35B\-A3B–73\.3$34\.06–Kimi\-K2\.6–71\.7$110\.70–DeepSeek\-V4\-Flash–68\.5$3\.16–MiniMax\-M2\.7–68\.2$24\.52–GLM\-4\.7–67\.7$63\.31–Gemini 3\.1 Flash Lite–64\.5$6\.85–DeepSeek\-R1–64\.3$57\.38–*Embedding Models \(ranked by score\)*Octen\-8B7\.6B77\.2$0\.108–Qwen3\-E\-8B7\.6B77\.0$0\.108[78](https://arxiv.org/html/2608.12875#bib.bib38)Qwen3\-E\-4B4\.0B75\.9$0\.069[78](https://arxiv.org/html/2608.12875#bib.bib38)Nemotron\-8B7\.5B75\.3$0\.115[37](https://arxiv.org/html/2608.12875#bib.bib54)KaLM\-12B11\.8B74\.8$0\.158[80](https://arxiv.org/html/2608.12875#bib.bib37)Jina\-v5\-S596M74\.4$0\.034[3](https://arxiv.org/html/2608.12875#bib.bib81)SFR\-27\.1B73\.9$0\.136[47](https://arxiv.org/html/2608.12875#bib.bib39)Jina\-v5\-Nano212M73\.9$0\.010[3](https://arxiv.org/html/2608.12875#bib.bib81)F2LLM\-14B14\.0B73\.3$0\.215[79](https://arxiv.org/html/2608.12875#bib.bib76)GTE\-Qwen2\-7B7\.1B73\.1$0\.089[42](https://arxiv.org/html/2608.12875#bib.bib19)Linq\-Mistral7\.1B72\.8$0\.138[15](https://arxiv.org/html/2608.12875#bib.bib75)Qwen3\-E\-0\.6B596M72\.5$0\.018[78](https://arxiv.org/html/2608.12875#bib.bib38)EmbGemma\-300M308M72\.2$0\.008[39](https://arxiv.org/html/2608.12875#bib.bib56)F2LLM\-8B7\.6B72\.2$0\.122[79](https://arxiv.org/html/2608.12875#bib.bib76)F2LLM\-4B4\.0B71\.9$0\.079[79](https://arxiv.org/html/2608.12875#bib.bib76)F2LLM\-1\.7B1\.7B71\.7$0\.030[79](https://arxiv.org/html/2608.12875#bib.bib76)GTE\-Qwen2\-1\.5B1\.5B70\.8$0\.024[42](https://arxiv.org/html/2608.12875#bib.bib19)F2LLM\-0\.6B596M69\.9$0\.017[79](https://arxiv.org/html/2608.12875#bib.bib76)mE5\-L\-Inst560M69\.7$0\.005[72](https://arxiv.org/html/2608.12875#bib.bib18)BGE\-M3568M66\.5$0\.005[13](https://arxiv.org/html/2608.12875#bib.bib21)mE5\-L560M65\.9$0\.005[71](https://arxiv.org/html/2608.12875#bib.bib17)Arctic\-L\-v2568M65\.3$0\.005[77](https://arxiv.org/html/2608.12875#bib.bib74)mE5\-B278M64\.4$0\.002[71](https://arxiv.org/html/2608.12875#bib.bib17)mE5\-S118M63\.1$0\.001[71](https://arxiv.org/html/2608.12875#bib.bib17)E5\-Mistral\-7B7\.1B52\.2$0\.139[72](https://arxiv.org/html/2608.12875#bib.bib18)GritLM\-7B7\.2B51\.2$0\.138[48](https://arxiv.org/html/2608.12875#bib.bib20)Table 3:Complete model listingwith mean \(macro\) scores across 37 MTEB\(LLM\) tasks, total benchmark costs, and references\. LLM costs reflect actual API usage; embedding costs from H100 throughput benchmarking \($2\.49/hr\)\.
### B\.2Full Per\-Category Rankings

Table[4](https://arxiv.org/html/2608.12875#A2.T4)compares Pro with the best embedding in each category\. Table[5](https://arxiv.org/html/2608.12875#A2.T5)gives per\-category scores for all 36 models\. Pro ranks first overall largely because of retrieval; on classification, it ranks below ten embedding models\. The reasoning\-capable Gemini models rank far above the non\-reasoning Flash\-Lite, which places 31st of 36 overall\.

Table 4:Category\-level performance\(best LLM vs\. best embedding per category\)\.Δ\\Delta= best LLM−\-best embedding; bold = winner; task counts in parentheses\.Table 5:Complete per\-category resultsfor all 36 complete models, ranked by overall \(macro\) score\. Bold = best in column\.
### B\.3Per\-Task Scores \(Representative Models\)

Table[6](https://arxiv.org/html/2608.12875#A2.T6)reports individual task scores for 5 representative models \(2 LLMs, 3 embeddings\) across all MTEB\(LLM\) tasks, alongside the best embedding score per task\. The scores vary substantially by task\. For example, Pro is strongest on FQuAD retrieval, and SFR\-2 is strongest on SprintDuplicateQuestions\. This complementarity supports the hybrid\-pipeline recommendation in §[5](https://arxiv.org/html/2608.12875#S5)\.

TaskCat\.Gemini 3\.1 ProGemini 3 FlashQwen3\.6\-27BOcten\-8BQwen3\-E\-8BBest Emb\.*Classification \(Accuracy\)*AmazonCounterfactualClassificationCls84\.881\.290\.193\.092\.993\.0Banking77ClassificationCls85\.083\.180\.587\.387\.291\.6ImdbClassificationCls98\.097\.697\.497\.797\.897\.8MTOPDomainClassificationCls95\.596\.496\.998\.298\.199\.2MassiveIntentClassificationCls84\.985\.484\.685\.785\.888\.9MassiveScenarioClassificationCls79\.475\.976\.489\.389\.493\.0ToxicConversationsClassificationCls89\.690\.089\.291\.992\.094\.4TweetSentimentExtractionClassificationCls64\.263\.063\.277\.977\.878\.9*Clustering \(V\-measure\)*ArxivClusteringP2PClust60\.160\.354\.561\.962\.564\.1ArxivClusteringS2SClust62\.560\.141\.660\.561\.161\.5BiorxivClusteringP2PV2Clust62\.560\.849\.961\.665\.377\.3MedrxivClusteringP2PV2Clust52\.550\.648\.055\.955\.960\.7MedrxivClusteringS2SV2Clust54\.050\.342\.753\.355\.257\.5RedditClusteringP2PClust93\.990\.560\.178\.779\.880\.8StackExchangeClusteringP2PV2Clust42\.949\.046\.159\.759\.560\.7StackExchangeClusteringV2Clust91\.890\.273\.685\.886\.386\.3TwentyNewsgroupsClusteringV2Clust78\.779\.966\.368\.567\.673\.8*STS \(Spearmanρ\\rho\)*BIOSSESSTS89\.688\.785\.084\.482\.088\.8SICKRSTS86\.186\.876\.487\.988\.491\.1STS12STS82\.679\.577\.187\.387\.387\.4STS13STS91\.991\.590\.693\.994\.094\.6STS14STS90\.589\.388\.090\.991\.091\.4STS15STS94\.493\.991\.894\.394\.294\.5STS16STS89\.088\.587\.192\.592\.592\.9STS17STS94\.294\.191\.893\.693\.793\.7STS22v2STS73\.671\.772\.069\.568\.969\.5STSBenchmarkSTS92\.691\.789\.693\.293\.494\.9*PairClassification \(Avg\. Precision\)*LegalBenchPCPairCls72\.277\.471\.670\.070\.473\.6RTE3PCPairCls95\.496\.792\.985\.485\.785\.7SprintDuplicateQuestionsPCPairCls82\.283\.484\.699\.499\.699\.8TwitterURLCorpusPCPairCls82\.887\.685\.289\.490\.290\.2*Retrieval \(Recall@1\)*AILAStatutesRetr14\.55\.714\.223\.221\.823\.2FQuADRetrievalRetr92\.088\.096\.069\.068\.072\.0HC3FinanceRetrievalRetr71\.060\.057\.066\.059\.067\.0LegalBenchConsumerContractsQARetr86\.079\.088\.067\.070\.070\.0PublicHealthQARetr90\.049\.086\.081\.078\.085\.0TwitterHjerneRetrievalRetr33\.432\.433\.029\.928\.631\.5

Table 6:Per\-task scoresfor representative models across all 37 MTEB\(LLM\) tasks\. Bold = best in row \(incl\. best embedding\)\. Metric per category in italics\.
### B\.4Full Per\-Task Scores for All Models

Tables[7](https://arxiv.org/html/2608.12875#A2.T7)–[11](https://arxiv.org/html/2608.12875#A2.T11)provide the complete score matrix for all 36 models across every MTEB\(LLM\) task, presented one table per category so that each fits the page upright\. The tables share a common layout: models are rows, ordered by overall MTEB\(LLM\) score, with the ten LLMs listed first and the 26 embedding models below; the category’s tasks are columns, with its metric given in the caption\. The final column gives each model’s mean over the category, matching the corresponding column of Table[1](https://arxiv.org/html/2608.12875#S4.T1)\. Bold marks the best value in a column, whether a single task or the category mean\. Because rows are ordered by overall score rather than by category, this column is not monotonic: it shows where a model over\- or under\-performs its overall rank\. These tables extend Table[6](https://arxiv.org/html/2608.12875#A2.T6)to all models\.

Table 7:Per\-task scores: Classification \(Accuracy\)\.Table 8:Per\-task scores: STS \(Spearmanρ\\rho\)\.Table 9:Per\-task scores: Clustering \(V\-measure\)\.Table 10:Per\-task scores: Pair Classification \(AP / Accuracy\)\.Table 11:Per\-task scores: Retrieval \(Recall@1\)\.
### B\.5Benchmark Task Suite

Table[12](https://arxiv.org/html/2608.12875#A2.T12)lists all 37 tasks with languages, sample counts, and source citations\. All are held\-out subsets \(seed = 42\) derived from MTEB and MMTEB tasks, hosted on Hugging Face undermteb/llm\-eval\-\*; the exact dataset path and pinned revision for each task are listed in the released repository\. Table[13](https://arxiv.org/html/2608.12875#A2.T13)provides per\-task token budgets for cost estimation, broken down by split\.

TaskLang\.NCls\.MetricSourceClassification \(8 tasks\)ImdbClsen5002Acc\.[45](https://arxiv.org/html/2608.12875#bib.bib63)Banking77Clsen3k77Acc\.[11](https://arxiv.org/html/2608.12875#bib.bib57)AmazonCounterfactualClsen, de, ja8092Acc\.[54](https://arxiv.org/html/2608.12875#bib.bib65)MTOPDomainClsen, de, fr2k11Acc\.[41](https://arxiv.org/html/2608.12875#bib.bib66)MassiveIntentClsen, de, fr, ja4k60Acc\.[24](https://arxiv.org/html/2608.12875#bib.bib58)MassiveScenarioClsen, de, fr, ja3k18Acc\.[24](https://arxiv.org/html/2608.12875#bib.bib58)ToxicConversationsClsen5002Acc\.[8](https://arxiv.org/html/2608.12875#bib.bib64)TweetSentimentClsen5003Acc\.[49](https://arxiv.org/html/2608.12875#bib.bib10)Semantic Textual Similarity \(10 tasks\)STSBenchmarken500–Spearman[12](https://arxiv.org/html/2608.12875#bib.bib60)SICK\-Ren500–Spearman[46](https://arxiv.org/html/2608.12875#bib.bib61)STS12en500–Spearman[2](https://arxiv.org/html/2608.12875#bib.bib67)STS13en500–Spearman[2](https://arxiv.org/html/2608.12875#bib.bib67)STS14en500–Spearman[2](https://arxiv.org/html/2608.12875#bib.bib67)STS15en500–Spearman[1](https://arxiv.org/html/2608.12875#bib.bib68)STS16en500–Spearman[1](https://arxiv.org/html/2608.12875#bib.bib68)BIOSSESen100–Spearman[60](https://arxiv.org/html/2608.12875#bib.bib59)STS17en, de, es, fr1k–Spearman[12](https://arxiv.org/html/2608.12875#bib.bib60)STS22v2en, de, es, fr, ru, zh2k–Spearman[14](https://arxiv.org/html/2608.12875#bib.bib69)Clustering \(9 tasks\)RedditClustP2Pen1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)TwentyNewsgroupsV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)StackExchangeClustP2PV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)StackExchangeClustV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)ArxivClustP2Pen1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)ArxivClustS2Sen1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)BiorxivClustP2PV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)MedrxivClustP2PV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)MedrxivClustS2SV2en1k–V\-meas\.[49](https://arxiv.org/html/2608.12875#bib.bib10)Pair Classification \(4 tasks\)SprintDuplicateQuestionsPCen5002Avg\. Prec\.[57](https://arxiv.org/html/2608.12875#bib.bib73)TwitterURLCorpusPCen5002Avg\. Prec\.[36](https://arxiv.org/html/2608.12875#bib.bib70)LegalBenchPCen5002Avg\. Prec\.[31](https://arxiv.org/html/2608.12875#bib.bib62)RTE3PCde, en, fr, it2k2Avg\. Prec\.[27](https://arxiv.org/html/2608.12875#bib.bib71)Retrieval \(6 tasks\)AILAStatutesen50 Q / 82 C–Recall@1[23](https://arxiv.org/html/2608.12875#bib.bib15)FQuADRetrievalfr100 Q / 269 C–Recall@1[19](https://arxiv.org/html/2608.12875#bib.bib77)HC3FinanceRetrievalen100 Q / 415 C–Recall@1[32](https://arxiv.org/html/2608.12875#bib.bib78)LegalBenchConsumerContractsQAen100 Q / 154 C–Recall@1[31](https://arxiv.org/html/2608.12875#bib.bib62)PublicHealthQAen100 Q / 172 C–Recall@1HF datasetTwitterHjerneRetrievalda77 Q / 262 C–Recall@1[69](https://arxiv.org/html/2608.12875#bib.bib72)

Table 12:MTEB\(LLM\) task suite \(37 tasks\)\.N = held\-out test samples \(summed over languages for multilingual tasks\); Q = queries, C = corpus documents\. Multilingual tasks are evaluated per language and averaged\. Held\-out subsets \(seed 42\) derived from MTEB and MMTEB tasks\([49](https://arxiv.org/html/2608.12875#bib.bib10);[23](https://arxiv.org/html/2608.12875#bib.bib15)\), hosted atmteb/llm\-eval\-\*\. Token counts: Table[13](https://arxiv.org/html/2608.12875#A2.T13)\.Table 13:Token budget per task\(GPT\-4o tokenizer; raw text\)\. Actual counts vary by±\{\\pm\}15–40% across model vocabularies\.\*Train split is the kNN reference corpus processed only by embedding models; LLMs process only the test split\. Corpus\-in\-context formatting adds∼\{\\sim\}20 tokens per document for LLM retrieval\.

## Appendix CAblation Studies

This section details the reduced\-thinking and few\-shot experiments and provides a per\-task retrieval breakdown\.

### C\.1Reduced\-Thinking Ablation Details

Our reduced\-thinking ablation uses Gemini 3 Flash withreasoning\_effort=low, which instructs the model to minimise internal chain\-of\-thought reasoning\. We evaluate on all 6 retrieval tasks and 3 representative classification tasks \(IMDB, Banking77, ToxicConversations\)\.

The reduction differs by task type: 54–94% on retrieval and 12–30% on classification\. This suggests the model adaptively allocates more reasoning to retrieval \(where documents must be read and compared\) than classification \(where the decision is more immediate\)\.

Reduced reasoning preserves or improves all six Flash retrieval scores and four of six cross\-family averages \(Figure[4](https://arxiv.org/html/2608.12875#S4.F4)b\)\. We hypothesise that excessive reasoning introduces second\-guessing: the model considers multiple interpretations of a document’s relevance when a more direct reading would suffice\.

On classification, all three tasks show\|Δ\|<1\.0\|\\Delta\|<1\.0, indicating no measurable benefit from additional reasoning in this ablation\. This result is consistent with the supervision asymmetry between kNN and zero\-shot classification\.

Table 14:Reduced\-thinking ablation\(Gemini 3 Flash withreasoning\_effort=lowvs\. default\) on the MTEB\(LLM\) retrieval tasks\. Think↓\\downarrow= reduction in thinking tokens vs\. default\. Reducing thinking by 54–94% improves all six retrieval scores in this ablation\.Table 15:Few\-shot classification ablation\(Flash, 5 in\-context examples vs\. zero\-shot\)\.Bold= best score per task across all methods\. Five\-shot prompting matches zero\-shot performance on the 2–3 class tasks and lowers Banking77 performance, where the prompt contains five examples for 77 labels\.
### C\.2Per\-Task Retrieval Analysis

The six retrieval tasks in our benchmark span qualitatively different retrieval challenges; per\-task scores for all 36 models are in Table[11](https://arxiv.org/html/2608.12875#A2.T11)\. We provide a per\-task breakdown of the aggregate retrieval advantage \(\+\+8\.5\) reported in §[4\.2](https://arxiv.org/html/2608.12875#S4.SS2)\. Pro leads five retrieval tasks, and Octen\-8B leads AILAStatutes\.

#### FQuAD \(French reading\-comprehension QA\)\.

Answering requires reading French passages in context and matching the supporting one\. Pro \(92\.0\) leads the best embedding \(72\.0\) by a wide margin, the largest LLM advantage in the suite\.

#### Consumer\-Contracts QA \(legal\)\.

Yes/no questions over consumer\-contract clauses\. Pro \(86\.0\) outperforms the best embedding \(70\.0\), consistent with the need to interpret the clause\.

#### Public\-Health QA\.

Pro \(90\.0\) edges the best embedding \(85\.0\); factual health QA is partly captured by dense retrieval, narrowing the gap\.

#### HC3\-Finance QA\.

Pro \(71\.0\) narrowly leads the best embedding \(67\.0\) on financial QA retrieval\.

#### TwitterHjerne \(Danish social media\)\.

Retrieving relevant tweets given a query\. Pro \(33\.4\) slightly outperforms embeddings \(best: 31\.5\); both handle keyword\-centric retrieval similarly\.

#### AILAStatutes \(legal statute retrieval\): where reasoning hurts\.

Matching case facts to relevant statutes\. The best embedding \(Octen\-8B, 23\.2\) outperforms Pro \(14\.5\): Pro over\-retrieves broadly associated provisions, while cosine similarity naturally produces a narrower, more precise ranking\.

## Appendix DPer\-Category Rankings and Figures

This section provides task\- and model\-level figures, followed by category\-level rankings\. Figure[6](https://arxiv.org/html/2608.12875#A4.F6)is the task\-level view of the paper’s central claim and is referenced from §[4\.2](https://arxiv.org/html/2608.12875#S4.SS2)\.

### D\.1Task\- and Model\-Level Figures

The figures below show task\-level performance, throughput, category profiles, and cross\-model correlations\.

Figure 6:Per\-task performance\.Horizontal bars show the full embedding score range \(min to max across 26 models\) for each MTEB\(LLM\) task, grouped by category\. Circle = best embedding; diamond = best LLM on that task, taken over all ten \(six different LLMs hold it across the suite, Gemini 3\.1 Pro on 21 of 37 tasks\)\. Where the diamond falls inside the bar, some embedding model already matches the best LLM: this holds on 7 of 8 classification tasks and 7 of 10 STS tasks, but on only 1 of 6 retrieval tasks\.Figure 7:Same\-hardware inference throughput\.Two open\-weight LLMs and seven representative embedding models \(118M–14B\) served on one H100 \(tokens/second, log scale\); the full 26\-model embedding throughput is in Table[18](https://arxiv.org/html/2608.12875#A8.T18)\. Even the slowest embedding runs∼\\sim2\.5×\\timesfaster than the fastest LLM; the fastest runs∼\\sim736×\\timesfaster\.Figure 8:Capability profiles across five task categories\.Gemini 3\.1 Pro \(red\), Qwen3\.6\-27B \(amber\), and Octen\-8B \(blue\)\. The LLMs lead on retrieval, while Octen\-8B leads on classification and remains competitive across categories\.![Refer to caption](https://arxiv.org/html/2608.12875v1/task_correlation_heatmap.png)Figure 9:Cross\-model task\-behaviour correlation\.Pearson correlations based on per\-task MTEB\(LLM\) scores\. LLMs and embedding models form visually distinct clusters: LLMs are strongly intercorrelated while embedding models cluster by architecture family\. Lower inter\-paradigm correlations are consistent with their different category profiles\.
### D\.2Per\-Category Model Rankings

Figures[10](https://arxiv.org/html/2608.12875#A4.F10)–[15](https://arxiv.org/html/2608.12875#A4.F15)rank all 36 evaluated models overall and within each of the five MTEB task categories\. They show category\-dependent performance: LLMs lead on retrieval, embedding models lead on classification, and the two paradigms are statistically indistinguishable on clustering, STS, and pair classification\. Flash\-Lite ranks near the bottom in every category, below the reasoning\-capable Gemini models\.

![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_Overall.png)Figure 10:Overall rankings\.Pro \(77\.6\) narrowly leads Octen\-8B \(77\.2\); the difference lies within statistical noise \(pp= 0\.85\)\. Flash\-Lite \(64\.5\) ranks near the bottom\.![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_Classification.png)Figure 11:Classification rankings\.SFR\-2 ranks first \(90\.8\); Pro scores 85\.2 and ranks below ten embedding models\.![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_Clustering.png)Figure 12:Clustering rankings\.Statistical tie; SFR\-2 \(66\.7\) edges Pro \(66\.6\)\. Flash\-Lite scores 21\.7\.![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_STS.png)Figure 13:STS rankings\.Qwen3\-E\-4B \(88\.8\) and Pro \(88\.5\) are essentially tied; no significant paradigm difference \(pp= 0\.74\)\.![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_PairClassification.png)Figure 14:Pair Classification rankings\.KaLM\-12B \(87\.1\) outperforms Pro \(83\.2\)\.![Refer to caption](https://arxiv.org/html/2608.12875v1/ranking_Retrieval.png)Figure 15:Retrieval rankings\.Pro leads the best embedding \(64\.5 vs\. 56\.0\) and ranks first on five of six tasks\.
### D\.3Full Retrieve\-then\-Rerank Matrix

Table[16](https://arxiv.org/html/2608.12875#A4.T16)reports the complete matrix summarised by Figure[3](https://arxiv.org/html/2608.12875#S4.F3)\(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\): all four first\-stage retrievers crossed with every cross\-encoder and LLM listwise reranker, on BEIR and BRIGHT\.

Table 16:Retrieve\-then\-rerank matrix\.Average nDCG@10 for first\-stage retrievers crossed with cross\-encoder and LLM listwise \(†\\dagger\) rerankers over 7 BRIGHT and 5 BEIR tasks\. Bold = best per row\. An LLM reranker improves Qwen3\-E\-8B on BRIGHT \(22\.3→\\to35\.1\); on BEIR, the embedding alone scores highest \(63\.1\)\.

## Appendix EQualitative Error Analysis

We present model predictions from the evaluation files to illustrate differences between LLM and embedding approaches\. The examples cover classification ambiguity, STS calibration, retrieval precision, and differences between Pro and Flash\.

### E\.1Classification: Over\-Interpretation vs\. Anchored Reference

Several classification errors involve ambiguous inputs for which the LLM infers an intent that differs from the dataset convention\. With kNN, an embedding classifier can anchor its prediction to similar labelled examples, which helps on fine\-grained tasks such as Banking77 and MassiveIntent\.

#### Banking77 \(77 classes\): “locate my card\.”

> Input:“How do I locate my card?” Pro reasoning:“The user is asking how to locate their card, indicating that they have misplaced it or it is missing\. This aligns with the intent for a lost or stolen card\.” Pro prediction:lost\_or\_stolen\_cardIncorrect \(gold:card\_arrival\) Flash prediction:card\_arrivalCorrect

Flash predicts the gold label; Pro predictslost\_or\_stolen\_card\. The LLM infers “locate” implies loss; Flash’s lower\-depth reasoning follows the surface\-level reading\. In the training data, “locate my card” aligns with delivery tracking\. The kNN classifier matches to examples like “Where is my card?” and correctly predictscard\_arrival\.

#### Banking77: “know when my card will arrive\.”

> Input:“Is there a way to know when my card will arrive?” Pro reasoning:“The user is asking for information regarding the timeframe or expected date for their card’s delivery, which aligns with thecard\_delivery\_estimatecategory\.” Pro prediction:card\_delivery\_estimateIncorrect \(gold:card\_arrival\) Flash prediction:card\_delivery\_estimateIncorrect \(gold:card\_arrival\)

Both models make the same error here, illustrating the ambiguity between “arrival” and “delivery estimate” without labelled context\. The kNN classifier uses the full 77\-class reference set to resolve such near\-neighbour cases\.

#### MassiveIntent \(60 classes\): “remind me about my schedule\.”

> Input \(en\):“remind me about my schedule for the afternoon” Pro reasoning:“The user is asking to be reminded about their schedule for the afternoon, which involves checking or querying their calendar events\.” Pro prediction:calendar\_queryIncorrect \(gold:calendar\_set\)

“Remind me” is an action verb \(set a reminder\) but the LLM interprets it as information\-seeking \(query\)\. This example illustrates the ambiguity between action and query intents\.

#### MassiveScenario \(German\): “wir brauchen rosa\.”

> Input \(de\):“wir brauchen rosa” \(we need pink\) Pro reasoning:“The utterance ’wir brauchen rosa’ \(we need pink\) implies adding an item \(’rosa’ / pink\) to a shopping list or note\-taking application\.” Pro prediction:listsIncorrect \(gold:iot\)

The LLM correctly reasons about the surface form but miscategorises the scenario:iotrefers to controlling physical smart\-home devices, not arbitrary colour references\.

> Input \(de\):“wie ist die einstellung für die helligkeit an meinem bildschirm” \(what is the brightness setting on my screen\) Pro reasoning:“The user is asking about the brightness setting on their screen, which involves controlling or checking the state of a device\. This falls under the IoT category\.” Pro prediction:iotIncorrect \(gold:general\)

Both errors reflect the fine\-grained scenario taxonomy:iotrefers to controlling physical smart\-home devices, not arbitrary brightness settings or colour references\. The LLM correctly identifies device\-related surface features but miscategorises the scenario\.

#### AmazonCounterfactual: hypothetical “would be great\.”

> Input:“Would be great to have on my tablet instead of carrying the book\.” Pro reasoning:“The review expresses a hypothetical situation \(’have on my tablet’\) that has not actually happened, indicated by the phrase ’Would be great’\. This describes an alternative condition and its potential consequence\.” Pro prediction:counterfactualIncorrect \(gold:not\-counterfactual\)

The phrase “would be” supports the LLM’s hypothetical reading\. The dataset convention labels forward\-looking wishes asnot\-counterfactual\. This may reflect a mismatch between a plausible linguistic interpretation and the dataset’s annotation convention\.

#### When LLMs succeed on classification\.

> Input:“I still have not received my new card, I ordered over a week ago\.” Pro prediction:card\_arrivalCorrect

In this less ambiguous example, the LLM matches the gold annotation\.

### E\.2STS: Calibration and Semantic Precision

The examples below show two kinds of disagreement between LLM scores and STS annotations: sensitivity to fine distinctions and higher scores for inferred mechanistic equivalence\.

#### STSBenchmark: agreement on a simple paraphrase\.

> S1:“open again, open anew”S2:“open again or anew\.” Pro:5\.0Gold:5\.0\|Δ\|\|\\Delta\|:0\.0

The LLM score matches the annotation for this near\-identical paraphrase\.

#### STSBenchmark: conservative scoring on ideological distinctions\.

> S1:“…a shift from a muslim identity to an islamist identity\.” S2:“…to change a muslim identity for an islamic identity\.” Pro:4\.5Gold:5\.0\|Δ\|\|\\Delta\|:0\.5

The LLM distinguishes “Islamist” \(political ideology\) from “Islamic” \(religious adjective\); the gold standard treats them as equivalent\. This linguistically defensible distinction reduces agreement with the benchmark annotation\.

#### BIOSSES: conservative scoring on biomedical paraphrases\.

> S1:“It has recently been shown that Craf is essential for Kras G12D\-induced NSCLC\.” S2:“It has recently become evident that Craf is essential for the onset of Kras\-driven non\-small cell lung cancer\.” Pro reasoning:“Both sentences state that Craf is essential for the development of Kras\-driven NSCLC\. Sentence 1 specifies the particular mutation \(Kras G12D\), whereas Sentence 2 is slightly more general\.” Pro:3\.8Gold:4\.0\|Δ\|\|\\Delta\|:0\.2

The LLM correctly notes the specificity difference \(“Kras G12D” vs\. “Kras\-driven”\) and downweights accordingly\. The gold annotation gives 4\.0, treating the two as highly similar; the LLM’s 3\.8 is slightly more conservative\.

#### BIOSSES: LLM over\-scores on mechanistic equivalence\.

> S1:“More recently, IDH mutations and resultant 2\-hydroxyglutarate \(2HG\) production in leukemia cells were found to impair TET2 function\.” S2:“It has also been recently reported that mutations of the isocitrate dehydrogenase genes IDH1 and IDH2 inhibit TET catalytic activity\.” Pro reasoning:“Both sentences describe the exact same biochemical pathway: mutations in IDH genes lead to the production of 2\-hydroxyglutarate \(2\-HG\), which in turn inhibits TET2 enzymatic activity\.” Pro:3\.8Gold:3\.2\|Δ\|\|\\Delta\|:0\.6

The LLM identifies a mechanistic relationship and scores the pair 3\.8, compared with the gold score of 3\.2\. Expert adjudication would be needed to determine whether this difference reflects model calibration or annotation criteria\.

### E\.3Retrieval: Reasoning Depth vs\. Geometric Precision

The retrieval examples show how joint query–document processing can help on reading\-comprehension tasks and how broader associations can hurt precise statute retrieval\.

#### FQuAD: reading comprehension in context\.

On French reading\-comprehension retrieval, the LLM reads each candidate passage in the prompt and selects the one that answers the query, reaching Recall@1 = 92\.0 versus 72\.0 for the best embedding\. The LLM processes the query jointly with the candidate passages, which may explain part of the observed advantage\.

#### AILAStatutes: where reasoning hurts\.

> Query:Legal case about a bank employee terminated for misconduct\. Pro:Returns 14 documents spanning broad legal concepts \(contract law, employment law, due process\)\. Gold: only 4 specific statutes\. The LLM’s associative reasoning retrieves plausible but non\-relevant provisions\. Octen\-8B:Returns a narrower, more precise set via cosine similarity with legal embeddings\.

In this example, Pro retrieves plausible but irrelevant provisions, while the embedding model produces a narrower ranking\.

#### TwitterHjerne: corpus\-in\-context precision\.

> Query \(Danish\):“Sønnen vil gerne lave \#pebernødder\. De par gange jeg har prøvet det, blev de kun OK\. Er der nogen, der kan anbefale en opskrift? \#twitterhjerne” \(“My son would like to make peppernuts\. The few times I’ve tried, they were only OK\. Can anyone recommend a recipe? \#twitterhjerne”\) Pro:Returns documents \[1, 2\]: exact match\.Recall@1 = 1\.0 on this query\. Best embedding:Recall@1 = 31\.5 over the full task\.

For this query with explicit topical keywords, corpus\-in\-context retrieval returns the correct tweet first\. The overall task\-level advantage for Pro \(33\.4 vs\. 31\.5 for embeddings\) is modest, reflecting that both paradigms handle keyword\-centric retrieval similarly\.

### E\.4Model Scale Effects: Flash vs\. Pro on the Same Input

#### Banking77: Flash succeeds where Pro over\-interprets\.

As shown above, Flash correctly predictscard\_arrivalfor “How do I locate my card?” while Pro predictslost\_or\_stolen\_card\. In this example, more reasoning does not improve the prediction\. Pro’s extended chain\-of\-thought introduces an interpretive step \(“locate implies misplacement”\) that Flash’s lighter\-weight reasoning bypasses\. Aggregate scores provide context: Pro remains slightly stronger than Flash on classification overall\.

#### Retrieval scale effects\.

On FQuAD reading\-comprehension retrieval, all three Gemini models outscore the best embedding, with a clear quality ladder:

- •Pro \(Recall@1 = 92\.0\): reads and matches passages precisely\.
- •Flash \(Recall@1 = 88\.0\): captures most of the advantage\.
- •Flash\-Lite \(Recall@1 = 84\.0\): still well above embeddings, even without reasoning\.
- •Best embedding \(Recall@1 = 72\.0\): limited to single\-vector matching\.

Flash\-Lite already outperforms the best embedding on this task; Pro adds a modest gain at much higher cost\. This result is consistent with the reduced\-thinking ablation \(§[4\.6](https://arxiv.org/html/2608.12875#S4.SS6)\)\.

#### Summary\.

The examples illustrate four patterns:

1. 1\.Ambiguous label boundaries: LLM predictions can differ from dataset conventions; kNN anchors decisions to labelled examples\.
2. 2\.STS calibration: model scores can differ from annotations when they attend to different semantic details\.
3. 3\.Retrieval breadth: joint processing can help reading comprehension but can also retrieve broadly related material\.
4. 4\.Model scale: the value of additional capacity or reasoning varies across examples and must be weighed against cost\.

Estimating the frequency of these patterns requires a systematic error analysis\.

## Appendix FExtended Related Work

### F\.1LLMs as Text Encoders and Judges

The boundary between generative LLMs and discriminative embedding models has blurred considerably\. LLM\-as\-a\-judge frameworks\([81](https://arxiv.org/html/2608.12875#bib.bib22)\)repurpose frozen LLMs for evaluation\.[66](https://arxiv.org/html/2608.12875#bib.bib23)find that zero\-shot LLM classification underperforms fine\-tuned models on standard benchmarks and propose chain\-of\-thought prompting to narrow the gap\. Fine\-tuned encoder models outperform zero\-shot frontier LLMs across the classification benchmarks tested by[9](https://arxiv.org/html/2608.12875#bib.bib44), with margins of 5 to over 75 F1 points depending on task granularity\. This is consistent with our 5\.6\-point advantage for kNN\-equipped embedding pipelines\. The LOFT benchmark\([38](https://arxiv.org/html/2608.12875#bib.bib24)\)evaluates long\-context LLMs on retrieval by placing entire corpora in the prompt \(corpus\-in\-context prompting\); our retrieval evaluation adopts the same protocol\.

On the representational side, several lines of work extract embeddings from LLMs\. SGPT\([50](https://arxiv.org/html/2608.12875#bib.bib25)\)uses weighted mean pooling over GPT decoder states\.[33](https://arxiv.org/html/2608.12875#bib.bib26)use in\-context learning to extract sentence embeddings from decoder\-only LLMs, while[7](https://arxiv.org/html/2608.12875#bib.bib27)adapt decoder\-only LLMs into sentence encoders\.[61](https://arxiv.org/html/2608.12875#bib.bib28)show that repeating the input can improve these embeddings by enabling effective bidirectional attention\. GritLM\([48](https://arxiv.org/html/2608.12875#bib.bib20)\)goes further, unifying generation and representation in a single model that achieved strong performance on both MTEB and generative benchmarks at the time of its release\.

These methods combine elements of both paradigms\. At inference time, they produce fixed\-length vectors compared via cosine similarity, so our cost accounting treats them as embedding models\. Several of our 26 evaluated models \(e\.g\., E5\-Mistral, GritLM\) are built on LLM backbones and incorporate LLM\-derived representations\.

HUME\([5](https://arxiv.org/html/2608.12875#bib.bib11)\)also benchmarks LLMs as annotators on MTEB tasks and finds that they fall short of human judgment on reranking\. This complements our finding that zero\-shot LLM classifiers underperform embedding pipelines with kNN access to labelled references\. Our study adds a controlled comparison on identical tasks, data, and metrics, together with exact cost accounting\.

### F\.2Frontier LLMs and Open\-Weight Reasoning Models

LLMs have advanced rapidly since GPT\-4\([52](https://arxiv.org/html/2608.12875#bib.bib36)\)\. OpenAI’s o1\([53](https://arxiv.org/html/2608.12875#bib.bib40)\)demonstrated extended chain\-of\-thought reasoning at inference time using reinforcement learning\. At the time of our evaluation, closed\-source models with controllable reasoning included Gemini 3\.1 Pro, GPT\-5\([58](https://arxiv.org/html/2608.12875#bib.bib1)\), and Claude Opus 4\.6\([4](https://arxiv.org/html/2608.12875#bib.bib2)\)\. These models support extended thinking tokens billed separately from output; newer models have since appeared \(e\.g\. GPT\-5\.4\)\. Open\-weight alternatives include Llama 3\([30](https://arxiv.org/html/2608.12875#bib.bib3)\), Qwen3\([76](https://arxiv.org/html/2608.12875#bib.bib4)\), and DeepSeek\-R1\([16](https://arxiv.org/html/2608.12875#bib.bib8)\)\. DeepSeek\-R1 shows that reinforcement\-learning\-based training can elicit strong reasoning at lower inference cost\. Gemma 3\([26](https://arxiv.org/html/2608.12875#bib.bib9)\)is a compact multilingual model family\.

We evaluate ten LLMs across six families \(§[3](https://arxiv.org/html/2608.12875#S3)\), including DeepSeek\-R1 and open\-weight Qwen3\.6, GLM, Kimi, and MiniMax models alongside Gemini 3\. The cost and task\-level patterns recur across this set, although their magnitude varies by model\. GPT\-5\([58](https://arxiv.org/html/2608.12875#bib.bib1)\)and Claude Opus 4\.6\([4](https://arxiv.org/html/2608.12875#bib.bib2)\)remain to be evaluated\. The same evaluation framework can accommodate these and future models\.

### F\.3Cross\-Encoders and Late\-Interaction Retrieval

Cross\-encoder rerankers \(models that jointly encode query and document to produce a relevance score\) and late\-interaction models such as ColBERT\([35](https://arxiv.org/html/2608.12875#bib.bib53)\)occupy a middle ground between bi\-encoder speed and LLM reasoning depth\. We evaluate this middle ground directly \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\) by crossing four first\-stage retrievers with cross\-encoder and LLM listwise rerankers on BEIR and BRIGHT\([65](https://arxiv.org/html/2608.12875#bib.bib41);[74](https://arxiv.org/html/2608.12875#bib.bib5)\)\. Reranking improves reasoning\-heavy retrieval, while the strongest embedding alone remains best on semantic retrieval\.

[43](https://arxiv.org/html/2608.12875#bib.bib46)compare RAG with long\-context LLMs on QA tasks\. Long\-context LLMs achieve higher quality at greater token cost, while a Self\-Route hybrid retains 93–100% of their quality at 38–61% of the token cost, depending on the model\. This result supports the hybrid recommendation in §[5](https://arxiv.org/html/2608.12875#S5)\.

## Appendix GExtended Limitations

We expand on the six limitations summarised in §[5](https://arxiv.org/html/2608.12875#S5.SS0.SSS0.Px2)\.

#### \(i\) LLM coverage\.

The ten evaluated LLMs span six families, dense and mixture\-of\-experts architectures, and reasoning and instruct variants\. They provide a snapshot of a fast\-moving frontier\. At current public rates, the Pareto frontier includes Gemini 3\.1 Pro and the leading embeddings; the lowest\-cost LLM evaluated scores below the best embedding\. Closed APIs not yet covered \(GPT\-5\([58](https://arxiv.org/html/2608.12875#bib.bib1)\)and Claude Opus 4\.6\([4](https://arxiv.org/html/2608.12875#bib.bib2)\)\) may have different cost–capability profiles, so the numerical conclusions should be updated as new evaluations become available\. The released evaluation pipeline makes this straightforward to extend as new models are released\.

#### \(ii\) Supervision asymmetry\.

Classification uses labelled\-reference kNN for embeddings and zero\-shot prompting for LLMs\. This setup reflects deployments with scarce labelled data and gives the two pipelines different supervision\. Five examples preserve performance on the small\-label tasks and reduce Banking77 performance \(§[4\.7](https://arxiv.org/html/2608.12875#S4.SS7)\)\.

#### \(iii\) Small\-corpus retrieval\.

The corpus\-in\-context protocol places entire corpora \(82–415 documents\) in the LLM prompt, a setting that grants the LLM three structural advantages: the corpus fits entirely in context, prompt caching amortises input cost across queries, and the model can reason globally over the full document set\. Each is a property of the protocol rather than a measured effect; we did not vary corpus size, so we do not quantify how retrieval quality would change at larger scale\. At BEIR or production scale, placing the full corpus in context is generally impractical; an indexed first stage is needed, so the small\-corpus cost gap is a lower bound for this protocol\. We complement corpus\-in\-context retrieval with bi\-encoder, cross\-encoder, and LLM\-reranker comparisons on BEIR and BRIGHT \(§[4\.3](https://arxiv.org/html/2608.12875#S4.SS3)\)\. Reranking benefits reasoning\-heavy retrieval; the strongest embedding retains the lead on semantic retrieval\([65](https://arxiv.org/html/2608.12875#bib.bib41);[74](https://arxiv.org/html/2608.12875#bib.bib5)\)\. nDCG@kkresults for the small\-corpus setting are available in the released data files and preserve the paradigm\-level ordering\.

#### \(iv\) Reasoning\-control granularity\.

Reasoning controls differ by provider: Gemini’sreasoning\_effort=lowprovides a soft hint, while the open models support fully disabled reasoning\. We use both settings\. Reduced reasoning preserves or improves retrieval for four of six models across five families \(Figure[4](https://arxiv.org/html/2608.12875#S4.F4)b\); token accounting varies by provider\.

#### \(v\) Throughput comparison\.

We serve two open\-weight LLMs \(Qwen3\.6\-27B and Qwen3\.6\-35B\-A3B\) and the embedding models on the same H100, removing API rate limits and hardware choice as confounds\. The resulting 2\.5–736×\\timesgap is consistent with the difference between autoregressive decoding and a single encoder pass\.

#### \(vi\) Task weighting sensitivity\.

Our aggregate metric weights all MTEB\(LLM\) tasks equally regardless of dataset size, task difficulty, or practical importance\. Alternative weighting schemes \(by dataset size, normalised z\-scores, or task\-family weighting\) may shift aggregate conclusions\. Aggregate numbers should therefore be interpreted alongside the reported category\- and task\-level results\. Alternative aggregation methods such as the Borda count used by MMTEB\([23](https://arxiv.org/html/2608.12875#bib.bib15)\)are applicable to MTEB\(LLM\) and may produce different paradigm orderings\.

## Appendix HCost and Throughput Details

This section reports token usage, embedding throughput, cost sensitivity, and the cost methodology underlying §[4\.4](https://arxiv.org/html/2608.12875#S4.SS4)–[4\.5](https://arxiv.org/html/2608.12875#S4.SS5)\.

### H\.1Detailed LLM Token Usage

Table[17](https://arxiv.org/html/2608.12875#A8.T17)provides the complete token\-level breakdown for each LLM across all MTEB\(LLM\) tasks\. For each model, we report input, standard\-output, cached, reasoning, and total tokens, together with the cost decomposition\. Gemini reports reasoning separately, whereas OpenRouter includes it in the completion count\. The reasoning\-heavy LLMs generate 8–26M reasoning tokens \(Qwen3\.6\-27B 26M, GLM\-4\.7 25M, Kimi\-K2\.6 22M, Gemini 3 Flash 14M, Pro 9M\), compared with 1–3M standard\-output tokens\. Reasoning accounts for 28–81% of total inference cost across these models: 81% for Qwen3\.6\-27B and 28% for DeepSeek\-V4\-Flash\. The non\-reasoning Flash\-Lite costs $6\.85 and ranks below most embeddings\.

Table 17:Detailed LLM token usage and costacross all MTEB\(LLM\) tasks\. Innc= non\-cached input \(cached billed at 10% of input rate\); Think = reasoning tokens \(billed at output rate\); Out\+Th combines standard output and thinking cost\.
### H\.2Embedding Throughput and Cost

Table[18](https://arxiv.org/html/2608.12875#A8.T18)reports the throughput \(tokens/second\) and derived cost per million tokens for each of the 26 embedding models, benchmarked on a single NVIDIA H100 80GB HBM3 GPU at sequence length 512 with maximum batch size\. Costs range from $0\.00016/MTok \(mE5\-small, 118M parameters, 4\.3M tok/s\) to $0\.047/MTok \(F2LLM\-v2\-14B, 14\.7K tok/s\)\. Total benchmark cost spans 291×\\timesacross the embedding models, compared with a 1,431×\\timesgap between the best embedding and Gemini 3\.1 Pro\.

Each model’s total benchmark cost uses its own tokenizer’s token count \(4\.4–5\.5M tokens across vocabularies\), avoiding tokenizer mismatch\.

Table 18:Embedding throughputon a single NVIDIA H100 80GB \(median tokens/s over the benchmark\), per\-MTok cost at $2\.49/hr spot, and mean \(macro\) MTEB\(LLM\) score\.
### H\.3Cost Sensitivity Analysis

Table[19](https://arxiv.org/html/2608.12875#A8.T19)shows how the LLM\-to\-embedding cost ratio varies under alternative hardware and pricing assumptions\. We consider five scenarios for embedding inference:

1. 1\.H100 spot\($2\.49/hr, Lambda Labs, March 2026\): baseline, 1,431×\\times
2. 2\.H100 on\-demand\($3\.99/hr\): higher price, 893×\\times
3. 3\.A100 spot\($1\.49/hr, est\. 1\.5×\\timesslower\): 1,594×\\times
4. 4\.L4 spot\($0\.49/hr, est\. 3×\\timesslower\): much lower throughput, 2,424×\\times
5. 5\.Commercial API\(e\.g\. $0\.10/MTok\): fixed per\-token pricing, 338×\\times

In all five scenarios, the cost ratio exceeds 300×\\times\.

Table 19:Cost sensitivity analysis\.LLM\-to\-embedding cost ratio under alternative hardware and pricing scenarios\. Costs compare Octen\-8B with Gemini 3\.1 Pro at fixed API pricing; ratios range from 338–2,424×\\times\.#### Per\-query economics and training amortisation\.

The benchmark\-level ratios translate directly to per\-query terms: both paradigms answer the same queries, so dividing by the query count rescales both sides equally and leaves the 29–1,431×\\timesrange unchanged\. Two effects widen the gap further under sustained deployment\. First, embedding training is a one\-time cost amortised over the deployment lifetime \(and, for public checkpoints, across all users\), so its per\-query contribution vanishes at scale, whereas LLM inference cost recurs in full on every query\. Second, document embeddings are computed once and reused across queries, while corpus\-in\-context reading repeats per query and is only partially offset by prompt caching\. The benchmark\-pass comparison is therefore conservative: it charges embeddings their full encoding cost while granting the LLM cached\-input rates\.

### H\.4Detailed Cost Methodology

#### Embedding cost pipeline\.

For each model, we tokenize all MTEB\(LLM\) tasks with its HuggingFaceAutoTokenizer\. We then benchmark maximum\-throughput processing on an NVIDIA H100 80GB at sequence length 512, using the largest batch that fits in memory\. Cost is\(tokens/106\)×\(rGPU/T\)\(\\text\{tokens\}/10^\{6\}\)\\times\(r\_\{\\text\{GPU\}\}/T\), whererGPU=$2\.49r\_\{\\text\{GPU\}\}=\\$2\.49/hr \(Lambda Labs spot pricing, March 2026\) andTTis sustained throughput in tokens/hour\.

Token counts range from 4\.4M \(Gemma\-based tokenizers\) to 5\.5M \(Mistral\-based\), a 26% range driven by vocabulary differences\. Using a single proxy tokenizer \(e\.g\., GPT\-2 at 6\.75M tokens\) would overestimate embedding costs by 22–54%\.

#### LLM cost pipeline\.

Token usage is extracted from each provider’susage\_statsin every task’s JSON result file\. For Gemini,input\_tokens=prompt\_token\_count\(including cached tokens\),output\_tokens=candidates\_token\_count\(excluding reasoning\), andcached\_tokens=cached\_content\_token\_count\. Total tokens equal input plus output plus reasoning\. For the open models served via OpenRouter, the completion count already includes reasoning andtotal= input \+ output, with reasoning reported separately\. We bill every generated token,total−input\\text\{total\}\-\\text\{input\}, at the output rate\. This equals output plus reasoning for Gemini and the reasoning\-inclusive completion count for open models\. Each model’s reasoning share comes from its usage statistics \(Figure[4](https://arxiv.org/html/2608.12875#S4.F4)\)\. For multilingual tasks \(STS17, STS22v2, RTE3\), per\-language entries report different token counts and are summed; for classification tasks with multiple entries per split, all entries share identicalusage\_statsand are counted once\.

Gemini API prices per MTok input/output are $0\.25/$1\.50 for Flash\-Lite, $0\.50/$3\.00 for Flash, and $2\.00/$12\.00 for Pro; cached input is billed at 10%\. Open models use OpenRouter public rates \(March/June 2026\)\.

#### Throughput measurement\.

We serve both paradigms on the same NVIDIA H100 and measure sustained tokens/second\. Open\-weight LLMs \(Qwen3\.6\-27B, Qwen3\.6\-35B\-A3B\) run under vLLM \(BF16, tensor\-parallel = 1, 256 concurrent requests, 200/100 input/output tokens\); embedding models run batch inference on the same GPU\. Holding hardware fixed removes API rate limits and GPU choice as explanations for the measured gap\.

## Appendix IReproducibility

#### LLM prompt templates\.

Prompt design follows a consistent structure across all five task categories:

- •System prompt: one sentence describing the task type and output format\.
- •User prompt: task\-specific instruction, followed by the input text\(s\)\.
- •Output schema: a Pydantic model with fieldsreasoning\(free text, optional\) andoutput\(constrained to valid labels or numeric ranges\)\. JSON mode is enabled for all calls\.

Example system prompts:*Classification*: “You are a text classifier\. Return a JSON with your reasoning and the predicted label from the provided list\.”*Retrieval*: “You are a retrieval assistant\. Given a query and a set of numbered documents, identify the document IDs most relevant to the query\. Return a JSON list of IDs in ranked order\.”

#### Schema validation and failure handling\.

All LLM outputs are validated against the Pydantic schema after generation\. If validation fails \(e\.g\., invalid label or malformed JSON\), the prompt is retried with an appended correction instruction, up to twice per sample\. The final validation failure rate is<<0\.3%; failed samples receive a null prediction and are counted as incorrect\.

#### Token usage tracking\.

For each API call we log:prompt\_token\_count,candidates\_token\_count,cached\_content\_token\_count,total\_token\_count, andthoughts\_token\_countfromGenerateContentResponse\.usage\_metadata\. Thinking tokens are verified astotal−prompt−candidates\\texttt\{total\}\-\\texttt\{prompt\}\-\\texttt\{candidates\}to cross\-check againstthoughts\_token\_count; agreement was exact in all cases\. Pricing sources and throughput configurations are documented in §[H\.4](https://arxiv.org/html/2608.12875#A8.SS4)\.