Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing

# Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing
Source: [https://arxiv.org/html/2608.05857](https://arxiv.org/html/2608.05857)
11institutetext:GenAI Engineering, Pegasystems, Kraków, Poland
11email:marcin\.rozmus@pega\.com22institutetext:AI Lab, Pegasystems, Amsterdam, The Netherlands
22email:peter\.van\.der\.putten@pega\.com33institutetext:LIACS, Leiden University, Leiden, the Netherlands
33email:p\.w\.h\.van\.der\.putten@liacs\.leidenuniv\.nl###### Abstract

Retrieval\-Augmented Generation systems rely on similarity scores to retrieve relevant content, yet scores are not directly comparable across embedding models due to differing geometric properties, complicating model migration and limiting threshold reuse\. We study how similarity scores can be related by learning mappings between score distributions rather than embeddings\. We introduce Synthetic Query Probing, generating queries from documents to create controlled query–chunk pairs, enabling large\-scale, reference\-free analysis of cross\-model similarity behavior\. We evaluate the approach on multiple embedding configurations and learn score conversion functions using linear, isotonic, and quantile mappings\. Experiments on SciFact and a proprietary corpus show that while models largely agree on rankings, their absolute scores exhibit systematic distortions\. Learned mappings partially align these spaces and improve threshold portability, with isotonic regression performing best\. Our results highlight the need for cross\-model calibration and position Synthetic Query Probing as a scalable framework for analyzing embedding comparability\.

## 1Introduction

Retrieval\-Augmented Generation \(RAG\) has become a dominant architecture for grounding large language models in domain\-specific scientific or enterprise knowledge, either for direct end\-user access or as knowledge sources for agents\[[8](https://arxiv.org/html/2608.05857#bib.bib11),[5](https://arxiv.org/html/2608.05857#bib.bib13),[10](https://arxiv.org/html/2608.05857#bib.bib14)\]\. At the core of every RAG pipeline lies vector similarity search: a user query is embedded, compared against a corpus of pre\-embedded document chunks, and the most similar chunks are selected as context for generation\.

A central assumption in this process is that similarity scores are comparable across embedding models\. In practice, however, these scores are produced by models with differing geometric properties, dimensionalities, and training objectives\. As a result, similarity scores are not directly comparable across models, configurations, or corpora, and thresholds calibrated for one setting do not transfer reliably to another\. This complicates model migration in RAG systems, which can happen frequently in practice, and limits our ability to reason systematically about relationships between embedding spaces\. While prior work extensively studies embedding quality and downstream performance, it generally treats similarity scores as model\-specific and does not address their comparability across models\. Consequently, whether similarity spaces can be systematically related and thresholds transferred remains largely unexplored, despite its practical importance for model migration\.

In this work, we investigate how similarity scores from different embedding models can be systematically related, and whether mappings between their similarity spaces can be learned\. To our knowledge we are the first to explicitly model and empirically analyze cross\-model similarity score distributions and their implications for threshold transfer in RAG systems\. We introduce*Synthetic Query Probing \(SQP\)*, a scalable, reference\-free methodology in which queries are automatically generated from document chunks with varying levels of relatedness\. This allows us to construct controlled query–document pairs and obtain corresponding similarity scores across multiple embedding models, enabling direct cross\-model comparison\. We evaluate SQP on four embedding configurations, analyzing similarity distributions and learning cross\-model conversion functions\. This study is making the following contributions:

1. \(i\)Problem formulation\.We formalize the lack of similarity score comparability across embedding models as an underexplored challenge, limiting our fundamental understanding of embedding spaces and impacting practical concerns such as threshold calibration and model migration\.
2. \(ii\)Synthetic Query Probing\.We introduce a scalable, reference\-free method for analyzing similarity behavior across embedding models by generating controlled query–document pairs without human annotation\.
3. \(iii\)Cross\-model similarity analysis\.Using this method, we empirically characterize how similarity distributions differ across models and dimensionalities, showing systematic distortions despite preserved ranking structure\.
4. \(iv\)Score calibration and transfer\.We show that these distortions can be modeled using learned transfer functions, and provide practical methods, including isotonic regression and quantile mapping, for converting similarity scores and thresholds across models\.

## 2Related Work

Research on evaluating text embedding models has largely followed two lines: benchmarking downstream task performance and directly analyzing embedding spaces\. Neither addresses whether the similarity distributions used to calibrate retrieval thresholds are comparable across models and corpora\.

### 2\.1Benchmarking Embeddings for Down\-Stream Task Performance

Nowadays, there are very mature benchmarks available that evaluate embeddings across a wide range of tasks, such as BEIR\[[13](https://arxiv.org/html/2608.05857#bib.bib10)\]and MTEB\[[9](https://arxiv.org/html/2608.05857#bib.bib4)\]\. As of June 2026, MTEB covers over 1000 languages, 130 tasks and \(partial\) results for over 400 models \(see also MMTEB\[[2](https://arxiv.org/html/2608.05857#bib.bib9)\]\)\. Embedding models are evaluated in terms of downstream performance for tasks such as clustering, classification, bitext mining, semantic textual similarity \(STS\), reranking, and retrieval\. Benchmark results evolve rapidly and vary substantially across tasks, languages, and domains, motivating both systematic model selection and comparison, and methods for understanding the impact of migration between embedding models\.

### 2\.2Beyond Task Benchmarking: Embedding Space Comparison

Recent work has shifted attention from aggregate benchmark rankings towards understanding the representational and behavioral properties of embeddings themselves, including representational similarity\[[7](https://arxiv.org/html/2608.05857#bib.bib19),[1](https://arxiv.org/html/2608.05857#bib.bib3)\], retrieval overlap across models\[[1](https://arxiv.org/html/2608.05857#bib.bib3)\], and robustness under meaning\-preserving perturbations\[[3](https://arxiv.org/html/2608.05857#bib.bib5),[4](https://arxiv.org/html/2608.05857#bib.bib6)\]\.

Caspari et al show that benchmarks alone are a weak guide for selecting embedding models in RAG\[[1](https://arxiv.org/html/2608.05857#bib.bib3)\]\. They quantify model similarity along representational similarity via Centered Kernel Alignment on the embedding geometry, and functional similarity via overlap \(Jaccard\) and ordering \(rank similarity\) of retrieved documents\. On five BEIR datasets, models cluster mainly by family yet diverge sharply in their top\-kkresults at smallkk, the regime that determines RAG quality, so comparable benchmark scores need not imply comparable retrieval\. Their work shows that representational and retrieval behavior, rather than benchmark rank, govern sustainability\. We complement this by characterizing the score distributions underlying threshold selection\.

A related concern is the stability of embeddings under meaning\-preserving variation\. The PTEB and HTEB benchmarks\[[3](https://arxiv.org/html/2608.05857#bib.bib5),[4](https://arxiv.org/html/2608.05857#bib.bib6)\]show that sentence encoders remain sensitive to paraphrasing and other semantic\-preserving transformations\. These results motivate our study: If similarity scores vary across both models and input perturbations, raw cosine values and their calibrated thresholds cannot be assumed to transfer across configurations\.

Perhaps the closest to our work is the study by Tacheny\[[12](https://arxiv.org/html/2608.05857#bib.bib20)\]\. To align embedding similarity scores with human judgments from MTEB\-STS\[[2](https://arxiv.org/html/2608.05857#bib.bib9)\], rather than performing preprocessing or tuning the embedding model they learn a monotonic calibration for the similarity function using a\.o\. linear and isotonic regression\.

### 2\.3Research Gap

While prior work evaluates task performance, representation similarity, retrieval overlap, or similarity to human judgment, it does not address the transferability of similarity thresholds across models\. To the best of our knowledge, no work has systematically compared cosine similarity score distributions across embedding models on multiple corpora and analyzed their implications for threshold transfer\.

## 3Methodology: Synthetic Query Probing

To facilitate practical adoption and reproducibility, we propose a deliberately simple, annotation\-free, scalable and configurable methodology and experimental protocol\. We model mappings between similarity scores rather than between embeddings themselves, avoiding assumptions about geometric alignment\.

Given a corpus𝒞\\mathcal\{C\}, it is partitioned into a set of chunks\{ck\}k=1N\\\{c\_\{k\}\\\}\_\{k=1\}^\{N\}using a chosen chunking strategy\. A uniformly random subset𝒞′⊆𝒞\\mathcal\{C\}^\{\\prime\}\\subseteq\\mathcal\{C\}is then sampled\. For each chunkc∈𝒞′c\\in\\mathcal\{C\}^\{\\prime\}, we generate a set of queries𝒬​\(c\)\\mathcal\{Q\}\(c\)using a large language model, covering varying degrees of semantic relatedness\. In our experiments, we consider the following query classes:

- •PARAPHRASE\(𝒬para​\(c\)\\mathcal\{Q\}\_\{\\text\{para\}\}\(c\)\): queries that can be answered directly and completely from the chunkcc\.
- •RELEVANT\(𝒬rel​\(c\)\\mathcal\{Q\}\_\{\\text\{rel\}\}\(c\)\): queries that are topically related toccbut require additional context beyond the chunk\.
- •IRRELEVANT\(𝒬irr​\(c\)\\mathcal\{Q\}\_\{\\text\{irr\}\}\(c\)\): queries drawn from unrelated domains and exhibiting no semantic connection tocc\.

These classes induce controlled relevance labels that enable evaluating score distributions and threshold behavior\. Without loss of generality, other synthetic labels of interest could be introduced, as long as the full similarity score range is covered\.

We focus on cosine similarity, which despite its limitations, is the dominant choice in production embedding\-based retrieval systems\[[11](https://arxiv.org/html/2608.05857#bib.bib16),[6](https://arxiv.org/html/2608.05857#bib.bib17),[16](https://arxiv.org/html/2608.05857#bib.bib18)\]but the method is open to using other distance metrics that address its limitations\[[16](https://arxiv.org/html/2608.05857#bib.bib18)\]\. Letℳ=\{mi\}i=1K\\mathcal\{M\}=\\\{m\_\{i\}\\\}\_\{i=1\}^\{K\}denote a set of embedding models\. For each modelmi∈ℳm\_\{i\}\\in\\mathcal\{M\}, we compute similarity scoressmi​\(q,c\)∈ℝs\_\{m\_\{i\}\}\(q,c\)\\in\\mathbb\{R\}for all query–chunk pairs\(q,c\)\(q,c\)withq∈𝒬​\(c\)q\\in\\mathcal\{Q\}\(c\), enabling direct cross\-model comparison and subsequent analysis, such as fitting similarity score mapping functions across models\.

## 4Experiments and Results

We evaluate Synthetic Query Probing across two corpora and four configurations, measuring per\-class similarity distributions, classification threshold precision, and cross\-model conversion accuracy\. All experiments follow the same framework, enabling comparison across models\.

### 4\.1Experimental Setup

We evaluate our approach on two corpora: an open scientific corpus and a large proprietary enterprise knowledge base\. Together they test whether score distributions and conversion functions are corpus\-dependent\.

The SciFact corpus contains 5,183 life\-science and biomedical abstracts\[[15](https://arxiv.org/html/2608.05857#bib.bib12)\]\. It is a homogeneous corpus with structured domain vocabulary, expected to produce cleaner semantic separation\. The enterprise corpus comprises 114,648 chunks from 33,492 internal and external Pegasystems documents, including product guides, API references, knowledge\-base articles, marketing materials, and community content, yielding broad cross\-topic similarity\.

To trade off coverage, cost, and stability of distributions we generate synthetic query probing data by sampling 100 chunks from each corpus, and use Claude Sonnet 4\.6 to generate 10 questions per chunk for each of the 3 relevance classes outlined in the previous section\. This process yields 3,000 question–chunk pairs per corpus with ground\-truth relevance labels\. For the embedding experiments, we use Amazon Titan Text Embeddings V2 at 256, 512 and 1024 dimensions as well as OpenAI text\-embedding\-ada\-002 with 1,536 dimensions\.111[https://www\.anthropic\.com/news/claude\-sonnet\-4\-6](https://www.anthropic.com/news/claude-sonnet-4-6),[https://openai\.com/index/new\-and\-improved\-embedding\-model/](https://openai.com/index/new-and-improved-embedding-model/),[https://docs\.aws\.amazon\.com/ai/responsible\-ai/titan\-text\-embeddings/](https://docs.aws.amazon.com/ai/responsible-ai/titan-text-embeddings/)

These question–chunk pairs, along with similarity scores, allow us to compare various embedding configurations, for instance by constructing ‘cross\-configuration mapping functions’\. These functions can be used in practice, for instance to map minimum similarity thresholds in retrieval from one model to another, but also provide deeper fundamental insight into the nature and behavior of models\. For each corpus, every question–chunk pair has all four cosine similarity scores, we can treat each pairwise mapping as a paired regression problem\. All 12 directional pairs are fitted: Titan 256→\\to512, 512→\\to256, 256→\\to1024, 1024→\\to256, 512→\\to1024, 1024→\\to512, 256→\\toAda, Ada→\\to256, 512→\\toAda, Ada→\\to512, 1024→\\toAda, Ada→\\to1024\. For each pair we evaluate three conversion approaches, linear and isotonic regression, and quantile \(CDF\) mapping\.

We also leverage the synthetic query probing data to carry out a ‘threshold analysis’\. For each corpus and each modelmi∈ℳm\_\{i\}\\in\\mathcal\{M\}independently, we perform a binary classification sweep over the similarity scoressmi​\(q,c\)s\_\{m\_\{i\}\}\(q,c\)\. The positive class collapses PARAPHRASE and RELEVANT queries,𝒬para​\(c\)∪𝒬rel​\(c\)\\mathcal\{Q\}\_\{\\text\{para\}\}\(c\)\\cup\\mathcal\{Q\}\_\{\\text\{rel\}\}\(c\), and the negative class is𝒬irr​\(c\)\\mathcal\{Q\}\_\{\\text\{irr\}\}\(c\)\. A queryqqis classified as positive at thresholdτ\\tauifsmi​\(q,c\)≥τs\_\{m\_\{i\}\}\(q,c\)\\geq\\tau\. We sweepT=500T=500evenly spaced candidates

𝒯mi=\{minmi\+k⋅maxmi−minmiT−1\|k=0,1,…,T−1\}\\mathcal\{T\}\_\{m\_\{i\}\}=\\left\\\{\\min\_\{m\_\{i\}\}\+k\\cdot\\frac\{\\max\_\{m\_\{i\}\}\-\\min\_\{m\_\{i\}\}\}\{T\-1\}\\;\\middle\|\\;k=0,1,\\ldots,T\-1\\right\\\}\(1\)
whereminmi=min\(q,c\)⁡smi​\(q,c\)\\min\_\{m\_\{i\}\}=\\min\_\{\(q,c\)\}s\_\{m\_\{i\}\}\(q,c\)is the observed minimum andmaxmi=max\(q,c\)⁡smi​\(q,c\)\\max\_\{m\_\{i\}\}=\\max\_\{\(q,c\)\}s\_\{m\_\{i\}\}\(q,c\)is the observed maximum of modelmim\_\{i\}on the evaluation set\. At eachτ∈𝒯mi\\tau\\in\\mathcal\{T\}\_\{m\_\{i\}\}we compute precision, recall, and F1\.

### 4\.2Per\-Class Similarity Distributions

Several observations emerge from the per\-class distribution analysis \(Table[1](https://arxiv.org/html/2608.05857#S4.T1)\)\. First, the Titan dimensionalities form a consistent gradient: as dimensionality decreases, all class means shift upward\. The effect is visible in the enterprise and SciFact corpora\. In contrast, Ada occupies a fundamentally different score space in both corpora\. Its similarity scores are compressed into a relatively narrow high range, with standard deviations that are two to six times smaller than those of the Titan variants\. As a result, the separation between classes is much smaller: the gap between IRRELEVANT and RELEVANT means, depending on the corpus, is two to four times smaller than for the Titan variants\.

Table 1:Cosine\-similarity statistics — SciFact and Enterprise corpora\.Despite these differences in scale and dispersion, class ordering is preserved across all configurations\. In every case, PARAPHRASE scores highest, followed by RELEVANT and then IRRELEVANT, and this ordering holds consistently across the mean, median, and all quartiles\. However, overlap between classes is most pronounced for Ada on the enterprise corpus, making threshold selection more challenging in that setting\. Notably, the maximum IRRELEVANT score for Ada exceeds the PARAPHRASE mean on the enterprise corpus, while on SciFact the Ada IRRELEVANT scores remain separated from the positive classes\. This reinforces the need for careful per\-corpus calibration when working in this score space\. Figure[1](https://arxiv.org/html/2608.05857#S4.F1)and Figure[2](https://arxiv.org/html/2608.05857#S4.F2)visualize these distributions as kernel density estimate \(KDE\) plots and box\-plots for all four configurations on each corpus\.

![Refer to caption](https://arxiv.org/html/2608.05857v1/x1.png)Figure 1:Cosine similarity distributions by semantic class for all four embedding configurations — SciFact corpus\.Top row: KDE density plots showing each configuration independently\. Titan variants show near\-perfect class separation with IRRELEVANT scores clustered near zero \(mean 0\.018 at 1024\-d, rising to 0\.061 at 256\-d\), while Ada compresses all classes into a 0\.626\-0\.930 band yet keeps IRRELEVANT fully disjoint from the positive classes\.Bottom row: Box\-plots confirming the upward IRRELEVANT shift at lower Titan dimensions and the cleaner class boundaries on SciFact compared to enterprise content\.![Refer to caption](https://arxiv.org/html/2608.05857v1/x2.png)Figure 2:Cosine similarity distributions by semantic class for all four embedding configurations \- Enterprise corpus\.Top row: KDE density plots showing each configuration independently\. Titan variants show clear class separation with IRRELEVANT scores clustered around 0\.2, while Ada compresses all classes into a narrow 0\.603\-0\.963 band\.Bottom row: Box\-plots confirming the upward shift in Titan scores at lower dimensions and Ada’s much tighter distributions\.
### 4\.3Precision\-First Threshold Analysis

For each configuration, we identify threshold operating points targeting three precision levels:≥\\geq0\.93 \(balanced\),≥\\geq0\.95 \(precision\-first\), and≥\\geq0\.97 \(high\-precision\)\. The binary target defines IRRELEVANT as negative \(0\) and PARAPHRASE \+ RELEVANT as positive \(1\)\.

The threshold analysis reveals a consistent pattern across both corpora\. On SciFact, achieving a precision of at least 0\.95 requires thresholds of 0\.063, 0\.090, and 0\.144 for Titan at 1024, 512, and 256 dimensions respectively\. On the enterprise corpus the same operating point demands 0\.393, 0\.400, and 0\.439 \- roughly six times higher in absolute terms, reflecting the denser overlap between classes in heterogeneous enterprise content\. In both cases, naively reusing a 1024\-d threshold at lower dimensionalities would lead to a noticeable drop in precision\.

Ada consistently requires a much higher absolute threshold than any Titan variant to reach the same precision level\. On SciFact, the 0\.95 operating point is at 0\.718 for Ada compared to 0\.063\-0\.144 for Titan\. On the enterprise corpus the gap is at 0\.821 compared to Titan’s 0\.393\-0\.439, reflecting the compressed nature of Ada’s score space regardless of corpus\.

The recall, however, is corpus\-dependent\. On SciFact, Ada achieves perfect recall \(1\.000\) against Titan’s 0\.981\-0\.996, because the clean separation between IRRELEVANT and positive classes in Ada’s score space on that corpus makes avoiding false positives trivial\. On the enterprise corpus the situation reverses, Titan variants retain higher recall at equivalent precision\. Titan’s wider class separation could be a more favorable precision\-recall trade\-off\.

Despite these differences, false positive counts remain comparable within each corpus\. At precision 0\.95, all configurations produce 96\-104 false positives on SciFact and 72\-79 on the enterprise corpus, indicating similar levels of noise rejection even though the underlying threshold scales differ substantially\. Figure[3](https://arxiv.org/html/2608.05857#S4.F3)shows precision, recall, and F1 as functions of the threshold for all four configurations on each corpus\.

Table 2:Threshold for the SciFact and Enterprise corpora at precision levels 0\.93, 0\.95, and 0\.97\. “FP” is the number of IRRELEVANT items above the threshold\.![Refer to caption](https://arxiv.org/html/2608.05857v1/x3.png)Figure 3:Precision, Recall, and F1 versus threshold for all four embedding configurations on the SciFact and Enterprise corpora\. The top row shows SciFact and the bottom row shows Enterprise\.
### 4\.4Cross\-Configuration Score Relationships

Since every row in the dataset has all four cosine similarity scores for the same question\-chunk pair, we can directly visualize the pairwise relationships\. Figure[4](https://arxiv.org/html/2608.05857#S4.F4)and[5](https://arxiv.org/html/2608.05857#S4.F5)show scatter and density plots for all six unique configuration pairs\.

![Refer to caption](https://arxiv.org/html/2608.05857v1/scifact_fig3_scatter.png)Figure 4:SciFact corpus: cross\-configuration score relationships for all six pairs\. Top row: Scatter plots colored by semantic class—PARAPHRASE \(orange\) in the upper\-right, IRRELEVANT \(red\) in the lower\-left\. The Titan cross\-dimension pairs show near\-linear relationships, reflecting the topical homogeneity of scientific abstracts\. The Titan–Ada pairs preserve the S\-shaped non\-linearity, but IRRELEVANT scores \(red\) are cleanly isolated in the lower\-left with no overlap into PARAPHRASE territory\. Bottom row: Hexagonally binned density plots confirming the sharper cluster boundaries and denser mapping curve on the scientific corpus\.![Refer to caption](https://arxiv.org/html/2608.05857v1/fig3_scatter.png)Figure 5:Enterprise corpus: cross\-configuration score relationships for all six pairs\. Top row: Scatter plots colored by semantic class—PARAPHRASE \(orange\) in the upper\-right, IRRELEVANT \(red\) in the lower\-left\. The Titan cross\-dimension pairs \(256–512, 256–1024, 512–1024\) show near\-linear relationships with tight scatter\. The Titan–Ada pairs exhibit a characteristic S\-shaped non\-linearity\. Bottom row: Hexagonally binned density plots revealing the concentration of observations along the mapping curve\.The scatter analysis reveals several clear patterns\. First, the Titan cross\-dimension pairs \(256–512, 256–1024, and 512–1024\) exhibit tight, approximately linear relationships with high correlation in both corpora, reflecting that these configurations are simply different dimensional variants of the same underlying model\. This linearity is even more pronounced on SciFact, where the topical homogeneity of scientific abstracts reduces pair scatter in comparison to the heterogeneous enterprise corpus\.

The Titan–Ada pairs display an S\-shaped non\-linearity in both corpora: at lower Ada scores \(around 0\.65\), the corresponding Titan scores are widely dispersed \(roughly 0\.0\-0\.4\), whereas at higher Ada scores \(around 0\.90\), the Titan scores converge within a narrower band \(approximately 0\.5\-0\.8\)\. This behavior provides a clear motivation for using non\-linear conversion methods\.

Despite these differences in geometry, class separation is consistently preserved across all projections\. In SciFact, the IRRELEVANT scores are cleanly isolated in the lower\-left of each plot with no overlap into positive class area, confirming that the underlying signal is strongly defined in the scientific corpus\. On the enterprise corpus, the three semantic classes form distinct clusters in every scatter plot, though some overlap is visible between IRRELEVANT and the positive classes in the Titan\-Ada projections\.

### 4\.5Conversion Model Evaluation

Table[3](https://arxiv.org/html/2608.05857#S4.T3)compares the three conversion methods across all twelve directional pairs on the SciFact and the enterprise corpora respectively\. For every directional pair and method we fit the corresponding parameters: the slope and intercept\(a,b\)\(a,b\)for linear OLS, the monotone step function \(knot set\) for isotonic regression, and the empirical percentile lookup for quantile mapping\. To keep this paper readable we omit the full per\-pair parameter sets and report only aggregate accuracy \(MAE and R2\) here\. Figure[6](https://arxiv.org/html/2608.05857#S4.F6)illustrates these fitted conversion functions for two representative directional pairs\.

Table 3:Conversion model accuracy for all twelve directional pairs on the SciFact and enterprise corpora\. Bold indicates the best method for each direction\.\(a\) SciFact corpus

\(b\) Enterprise corpus

![Refer to caption](https://arxiv.org/html/2608.05857v1/x4.png)Figure 6:Cross\-configuration conversion fits on the SciFact corpus for two representative dimension pairs, each shown with the linear OLS, isotonic regression, and quantile mapping conversion functions overlaid on similarity\-score pairs\.Cross\-dimension conversions prove to be highly accurate on both corpora\. Titan↔\\leftrightarrowTitan mappings reach R2values of at least 0\.967 on SciFact corpus \(MAE≤\\leq0\.035\) and at least 0\.980 on the enterprise corpus \(MAE≤\\leq0\.023\) across all methods\. In both cases the 512↔\\leftrightarrow1024 pair is the tightest \(R2= 0\.990 SciFact, 0\.996 enterprise\. MAE≈\\approx0\.010\-0\.019\), confirming that these two dimensionalities are nearly identical\.

By contrast, cross\-model conversions are inherently noisier, and their accuracy is markedly corpus\-dependent\. On the heterogeneous enterprise corpus, Titan↔\\leftrightarrowAda mappings achieve R2values of only 0\.820\-0\.918 with isotonic regression\. On the cleaner, topically homogeneous SciFact corpus the same conversions are substantially tighter\. The Titan↔\\leftrightarrowAda direction reaches R2= 0\.848\-0\.945 \(e\.g\. 0\.945 for 1024→\\toAda, MAE = 0\.014\), and the harder Ada→\\toTitan direction improves to 0\.85\-0\.89\. This gap mirrors the per\-class separation reported earlier: when IRRELEVANT queries are cleanly isolated, the cross\-model relationship is easier to fit\. The lower rows of SciFact figure make the difference visible: the characteristic S\-shaped non\-linearity of the Titan–Ada mapping is captured by isotonic regression but systematically missed by linear OLS, and the SciFact scatter is noticeably denser and better separated than the enterprise scatter\. In both corpora these results remain below the cross\-dimension figures but stay practically useful\. The residual noise reflects the fundamentally different architectures underlying the two models\.

Across all evaluated pairs and both corpora, isotonic regression consistently delivers the best performance, achieving the highest R2values alongside the lowest MAE\. This demonstrates its advantage for this type of conversion task\.

Finally, there is a notable asymmetry in error magnitude: the MAE for Ada→\\toTitan conversions is roughly three to four times higher than for the reverse Titan→\\toAda direction in both corpora \(e\.g\. 0\.073 vs\. 0\.019 on the enterprise corpus and 0\.060 vs\. 0\.014 on SciFact, for the 1024↔\\leftrightarrowAda pair\)\. This difference arises because Titan operates over a wider score range, which amplifies prediction errors in absolute terms even though the underlying fit quality \(R2\) is identical in both directions\.

## 5Discussion

Our experiments demonstrate the value of Synthetic Query Probing \(SQP\) as a reference\-free framework for analyzing similarity score behavior across embedding models\. By focusing on score distributions and threshold calibration, orthogonal to downstream ranking quality, we show that similarity functions exhibit approximately preserved ranking structure but systematic scale distortion across models and similarity ranges\.

A key finding is the existence of two regimes of conversion difficulty\. Cross\-dimension mappings within the Titan family are near\-lossless and largely corpus\-invariant \(R2≥\\geq0\.97\), reflecting their shared underlying representation\. In contrast, cross\-model mappings \(Titan↔\\leftrightarrowAda\) are inherently noisier, non\-linear, and corpus\-dependent, exhibiting an S\-shaped relationship that is best captured by isotonic regression\. Conversion accuracy is substantially higher on homogeneous corpora \(SciFact\) than on heterogeneous enterprise data, indicating that cross\-model comparability is not a fixed property of model pairs but depends on corpus characteristics\.

More broadly, conversion quality is strongly driven by class separability\. When relevant and irrelevant content are clearly separated, similarity mappings are tighter and more predictable; when overlap increases, mapping accuracy degrades, even for the same model pair\. This suggests that SQP\-derived separation statistics can serve as a prior indicator of expected conversion reliability before fitting any transfer function\.

We also observe an asymmetry in the sources of threshold variation\. Changing embedding dimensionality within a model family has a relatively small effect on optimal thresholds, whereas changing the corpus induces large shifts\. This implies that thresholds are primarily corpus\-dependent, and that per\-corpus calibration remains necessary even when the embedding model is held fixed\.

Across all experiments, isotonic regression consistently provides the best conversion accuracy\. Its ability to model monotonic but non\-linear relationships makes it particularly suitable for cross\-model calibration, whereas linear mappings suffice only for near\-linear regimes\. We also observe directional asymmetry: conversions from compressed score spaces \(e\.g\., Ada\) to wider ones \(e\.g\., Titan\) incur higher absolute error, reflecting differences in scale rather than fit quality\.

From a practical perspective, these findings imply that lightweight linear mappings may suffice for intra\-family changes \(e\.g\., dimensionality reduction\), but cross\-model migration requires non\-linear calibration and explicit validation on the target corpus\. More importantly, threshold reuse across corpora is considerably less reliable than reuse across model configurations\.

As the focus of this paper is to prove the method in principle, several limitations still qualify these results\. We evaluate only two corpora, and broader validation is required to assess generality, including additional sets of models and model families\. Synthetic queries may introduce bias, although prior work suggests such benchmarks are predictive of real retrieval behavior\[[14](https://arxiv.org/html/2608.05857#bib.bib1)\]\. Our analysis assumes normalized embeddings, static thresholds, and single\-stage retrieval without re\-ranking\. Finally, conversion functions are fitted on the full dataset and should be extended with proper train–test protocols for reusable calibration\.

Future work includes extending SQP to newer embedding models and a wider set of corpora, modeling conversion quality directly from corpus statistics and down\-stream task performance, using SQP for continuous monitoring of corpus drift to trigger re\-calibration, and also comparing different methods and models for question generation with SQP itself\. Together, these results position SQP as a practical and scalable tool for understanding and aligning embedding similarity spaces in evolving RAG systems\.

## 6Conclusion

Understanding the behavior and comparability of embedding\-based similarity functions is critical for both the scientific study of representation spaces and the reliable deployment of applications that rely on semantic embeddings such as Retrieval\-Augmented Generation systems\. We have shown that similarity scores produced by embedding models are not directly comparable, despite preserving relative ranking structure, and that their distributions exhibit systematic, model\- and corpus\-dependent distortions\. Using Synthetic Query Probing, we provide a scalable, reference\-free approach to characterize these differences and to learn transfer functions that partially align similarity spaces\. Our results demonstrate that isotonic regression offers an effective and practical solution for cross\-model calibration, highlighting the need for explicit threshold calibration when deploying and migrating RAG systems across models and corpora\.

\{credits\}

#### 6\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]L\. Caspari, K\. G\. Dastidar, S\. Zerhoudi, J\. Mitrovic, and M\. Granitzer\(2024\)Beyond benchmarks: evaluating embedding model similarity for retrieval augmented generation systems\.InProceedings of the Workshop Information Retrieval’s Role in RAG Systems \(IR\-RAG 2024\) co\-located with the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval \(SIGIR 2024\), Washington DC, USA, 07 18, 2024,F\. Petroni, F\. Siciliano, F\. Silvestri, and G\. Trappolini \(Eds\.\),pp\. 62–70\.Cited by:[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p2.2)\.
- \[2\]K\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzemiński, G\. Winata, S\. Sturua, S\. Utpala, M\. Ciancone, M\. Schaeffer, D\. Misra, S\. Dhakal, J\. Rystrøm, R\. Solomatin, Ö\. Çağatan, A\. Kundu, M\. Bernstorff, S\. Xiao, A\. Sukhlecha, B\. Pahwa, R\. Poświata, K\. K\. GV, S\. Ashraf, D\. Auras, B\. Plüster, J\. Harries, L\. Magne, I\. Mohr, D\. Zhu, H\. Gisserot\-Boukhlef, T\. Aarsen, J\. Kostkan, K\. Wojtasik, T\. Lee, M\. Suppa, C\. Zhang, R\. Rocca, M\. Hamdy, A\. Michail, J\. Yang, M\. Faysse, A\. Vatolin, N\. Thakur, M\. Dey, D\. Vasani, P\. Chitale, S\. Tedeschi, N\. Tai, A\. Snegirev, M\. Hendriksen, M\. Günther, M\. Xia, W\. Shi, X\. H\. Lu, J\. Clive, G\. K, M\. Anna, S\. Wehrli, M\. Tikhonova, H\. Panchal, A\. Abramov, M\. Ostendorff, Z\. Liu, S\. Clematide, L\. J\. V\. Miranda, A\. Fenogenova, G\. Song, R\. Bin Safi, W\. Li, A\. Borghini, F\. Cassano, L\. Hansen, S\. Hooker, C\. Xiao, V\. Adlakha, O\. Weller, S\. Reddy, and N\. Muennighoff\(2025\)MMTEB: massive multilingual text embedding benchmark\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 101715–101771\.Cited by:[§2\.1](https://arxiv.org/html/2608.05857#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p4.1)\.
- \[3\]M\. Frank and H\. Afli\(2026\-03\)PTEB: towards robust text embedding evaluation via stochastic paraphrasing at evaluation time with LLMs\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 2832–2851\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.130),ISBN 979\-8\-89176\-380\-7Cited by:[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p3.1)\.
- \[4\]M\. Frank and H\. Afli\(2026\)The Harder Text Embedding Benchmark \(HTEB\): beyond one\-dimensional static robustness\.External Links:2605\.28190,[Link](https://arxiv.org/abs/2605.28190)Cited by:[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p3.1)\.
- \[5\]Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. Wang\(2024\)Retrieval\-augmented generation for large language models: a survey\.External Links:2312\.10997,[Link](https://arxiv.org/abs/2312.10997)Cited by:[§1](https://arxiv.org/html/2608.05857#S1.p1.1)\.
- \[6\]V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih\(2020\-11\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6769–6781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§3](https://arxiv.org/html/2608.05857#S3.p3.5)\.
- \[7\]S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton\(2019\-09–15 Jun\)Similarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.Cited by:[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p1.1)\.
- \[8\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.05857#S1.p1.1)\.
- \[9\]N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers\(2023\-05\)MTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 2014–2037\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.148)Cited by:[§2\.1](https://arxiv.org/html/2608.05857#S2.SS1.p1.1)\.
- \[10\]A\. Plaat, M\. van Duijn, N\. Van Stein, M\. Preuss, P\. van der Putten, and K\. J\. Batenburg\(2026\-01\)Agentic large language models, a survey\.J\. Artif\. Int\. Res\.84\.External Links:ISSN 1076\-9757,[Document](https://dx.doi.org/10.1613/jair.1.18675)Cited by:[§1](https://arxiv.org/html/2608.05857#S1.p1.1)\.
- \[11\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP\-IJCNLP 2019, Hong Kong, China, November 3\-7, 2019,K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),pp\. 3980–3990\.External Links:[Document](https://dx.doi.org/10.18653/V1/D19-1410)Cited by:[§3](https://arxiv.org/html/2608.05857#S3.p3.5)\.
- \[12\]N\. Tacheny\(2026\)Calibrated similarity for reliable geometric analysis of embedding spaces\.External Links:2601\.16907,[Link](https://arxiv.org/abs/2601.16907)Cited by:[§2\.2](https://arxiv.org/html/2608.05857#S2.SS2.p4.1)\.
- \[13\]N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. Gurevych\(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),Cited by:[§2\.1](https://arxiv.org/html/2608.05857#S2.SS1.p1.1)\.
- \[14\]J\. van Elburg, P\. van der Putten, and M\. Marx\(2026\)Can we evaluate RAGs with synthetic data?\.InMachine Learning and Principles and Practice of Knowledge Discovery in Databases\. SynDAiTE workshop at ECML PKDD 2025\.,I\. Koprinska, J\. Mendes\-Moreira, and P\. Branco \(Eds\.\),Cham,pp\. 177–192\.External Links:ISBN 978\-3\-032\-19102\-1Cited by:[§5](https://arxiv.org/html/2608.05857#S5.p7.1)\.
- \[15\]D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. Hajishirzi\(2020\-11\)Fact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Online,pp\. 7534–7550\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.609)Cited by:[§4\.1](https://arxiv.org/html/2608.05857#S4.SS1.p2.1)\.
- \[16\]K\. You\(2025\)Semantics at an angle: when cosine similarity works until it doesn’t\.External Links:2504\.16318,[Link](https://arxiv.org/abs/2504.16318)Cited by:[§3](https://arxiv.org/html/2608.05857#S3.p3.5)\.