The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Summary
The paper compares large language models and embedding models across 37 tasks, finding that while aggregate performance is similar, embedding models are far cheaper and faster, supporting a division of labor for cost-efficiency.
View Cached Full Text
Cached at: 08/21/26, 08:11 PM
Paper page - The Embedder’s Dilemma: LLMs Are Better, but at What Cost?
Source: https://huggingface.co/papers/2608.12875
Abstract
Large language models and dedicated embedding models achieve nearly identical aggregate performance across diverse tasks, but embedding models are far cheaper and faster, supporting a division of labor by task type.
Should you replace yourtext-embeddingpipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26embedding models(118M to 14B parameters) on 37 tasks spanning classification,semantic textual similarity(STS),clustering, pair classification, andretrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavyretrieval,embedding modelslead on classification, and the two match onclustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU.Reasoning tokensaccount for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improveretrievalquality for most models in our ablation. ThePareto frontiercontains the leadingembedding modelsand one LLM, Gemini 3.1 Pro. These results support a division of labour: useembedding modelsfor similarity, classification, andclustering, and reserve LLMs for reasoning-intensiveretrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.12875
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.12875 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.12875 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.12875 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
This paper presents a cost-aware comparison of LLMs versus dedicated embedding models across 37 tasks, finding that the best LLM and embedding model are nearly tied on aggregate performance but LLMs are up to 1,431x more expensive and slower, leading to a recommended division of labor.
@tomaarsen: At 1400x cheaper, I know I'm sticking to embeddings (dense, sparse, multi-vector), plus hybrid (incl. bm25) and reranke…
A discussion on the cost-effectiveness of embeddings versus LLMs, referencing a study called 'embedder's dilemma' that finds LLMs outperform embedding models at significantly higher cost.
@v0xium: LLM Inference Engineering: Embedding Models Explained 1. An Embedding Model (EM) converts a chunk of text, or any other…
This article explains embedding models and their role in LLM inference, covering architectures, traffic profiles, and optimization techniques like quantization.
LLM-Based Embeddings for Program Analysis and Optimization
This paper presents the first application of program embeddings from LLMCompiler, an LLM pretrained on IR code, to program analysis and optimization tasks, achieving a 1.54% error rate in algorithm classification and competitive accuracy on heterogeneous device mapping.
Observational Equivalence of LLM and Human Annotation
This paper demonstrates that large language models (LLMs) achieve annotation quality equivalent to human experts, suggesting the main challenge is minimizing ambiguity in coding rules rather than choosing between human and machine coders.