Comparing Linear Probes with Mahalanobis Cosine Similarity
Summary
This paper extends empirical findings that the Mahalanobis cosine similarity (MCS) between linear probes linearly predicts out-of-distribution AUROC, and proves this relationship theoretically under Gaussian assumptions.
View Cached Full Text
Cached at: 06/23/26, 09:44 PM
Paper page - Comparing Linear Probes with Mahalanobis Cosine Similarity
Source: https://huggingface.co/papers/2606.19603
Abstract
The Mahalanobis cosine similarity provides a theoretically grounded method for comparing linear probes that correlates strongly with out-of-distribution performance metrics.
Linear probesare widely used in interpretability research and often compared by cosine similarity. TheMahalanobis cosine similarity(MCS) between two directions, which reweights the inner product bytest data covariance, is a natural task-aware refinement. Ying et al. (2026) report that a probe’s MCS to areference probetrained on the out-of-distribution (OOD) data near-perfectly linearly predicts the probe’s OOD AUROC (R^2 = 0.98). Here, we extend this empirical finding across models, layers, and concept domains, and prove this general phenomenon in closed form: For balanced classes whose projections are Gaussian, OOD AUROC and MCS to thereference probeare linear because both aresigmoid-shaped functionsof the probe’ssignal-to-noise ratio(SNR) on the test data. The theory also predicts when this linearity fails, which we verify empirically. MCS offers a theoretically grounded and empirically effective alternative to Euclidean cosine similarity for comparinglinear probes.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.19603
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.19603 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.19603 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.19603 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative
This paper demonstrates that mean-pooled cosine similarity is not length-invariant under anisotropic representations, showing it artificially inflates similarity with sequence length. It argues for using Centered Kernel Alignment (CKA) as a default metric to correct biases in cross-lingual and cross-representation analysis.
Cosine Similarity Is Not a Safety Property (18 minute read)
The article argues that cosine similarity in vector databases is a mathematical property that does not ensure factual accuracy, rendering retrieval systems susceptible to poisoning attacks where malicious documents can surpass legitimate ones in ranking.
Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing
Introduces Synthetic Query Probing (SQP), a reference-free method for learning mappings between similarity score distributions across embedding models, enabling threshold portability in RAG systems. Experiments show systematic score distortions and isotonic regression as the best calibration approach.
Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations
This paper systematically tests linear probes for deception detection in large language models, finding they fail under distributional shifts but style-augmented probes recover performance, and revealing that deception is encoded through distributed sub-threshold features.
Comparing embedding models with synthetic query probing [R]
The authors propose Synthetic Query Probing, a simple yet effective approach to compare different embedding models by comparing similarity match scores across models instead of raw embedding spaces. The paper illustrates non-linear relationships between models like Ada and Titan, offering practical guidance for swapping embedding models and setting retrieval thresholds.