Comparing embedding models with synthetic query probing [R]

Reddit r/MachineLearning Papers
embedding-models similarity-spaces synthetic-query-probing retrieval research

Summary

The authors propose Synthetic Query Probing, a simple yet effective approach to compare different embedding models by comparing similarity match scores across models instead of raw embedding spaces. The paper illustrates non-linear relationships between models like Ada and Titan, offering practical guidance for swapping embedding models and setting retrieval thresholds.

Say you want to swap out your embedding models, for instance from ADA to Titan. Are these embedding models comparable? How do similarity score ranges compare? Where to put a threshold for minimum match when doing retrieval? Or more from a research point of view how can we relate and fundamentally understand these embedding spaces better? This is what we aim to solve with Synthetic Query Probing, a fancy name for essentially (and intentionally) a very simple approach: embedding spaces are not directly comparable by definition, so compare similarity spaces instead, similarity match scores for pairs of content (synthetic question, chunk for instance) across multiple embedding models. For example, similarity scores of Titan models of different dimensionalities are related, whereas the relation between Titan and Ada scores is non-linear, with different ranges, see figure. https://preview.redd.it/eauhd4hdyiih1.png?width=4767&format=png&auto=webp&s=e424c836c48962928d9505cf747e7cd9fb0b719f See https://arxiv.org/pdf/2608.05857, Marcin Rozmus and Peter van der Putten. Similarity Spaces across Embedding Models with Synthetic Query Probing. Discovery Science 2026, October 5-9, 2026, Mainz, Germany
Original Article

Similar Articles

Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing

arXiv cs.CL

Introduces Synthetic Query Probing (SQP), a reference-free method for learning mappings between similarity score distributions across embedding models, enabling threshold portability in RAG systems. Experiments show systematic score distortions and isotonic regression as the best calibration approach.

How do you analyze the relative "strength" of probes? [R]

Reddit r/MachineLearning

The author asks how to analyze the relative 'strength' of probes in neural networks, discussing challenges such as limited vocabulary size and model capacity, and using an example from Google Gemini to illustrate failure cases.

Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG

arXiv cs.CL

This paper proposes replacing the proprietary GPT-3.5 metadata extractor in Multi-Meta-RAG with a lightweight, deterministic probe trained on hidden states of a small open-source model. The probe achieves 90.9% accuracy, outperforming GPT-3.5 (80.9%) and a substring baseline (88.0%), while avoiding allow-list drift and API costs.

Best-Arm Identification with Generative Proxy

arXiv cs.LG

This paper studies fixed-confidence best-arm identification where costly reward observations are paired with cheap but correlated proxy scores from generative models. It proposes the PROBE algorithm that uses control-variate adjustment and upper certificates on residual variance to achieve oracle-like sample complexity.