KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

# Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
Source: [https://arxiv.org/html/2606.04180](https://arxiv.org/html/2606.04180)
###### Abstract

Vision\-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems\. While these models are typically compared through downstream performance, such evaluations often do not explain how their representations differ structurally\. In this work, we study this problem through the task of*Contrastive Embedding Clustering*: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another\. We propose*Kernel Optimization for Discrepancy Analysis \(KODA\)*, a kernel\-based framework for contrastive representation comparison and alignment\. KODA constructs unified multimodal kernels through modality\-wise kernel composition and formulates discrepancy discovery as a constrained optimization problem that searches for coherent structures in one representation while suppressing coherence in a reference representation\. This yields interpretable discrepancy directions associated with specific sample subsets and modality interactions\. To scale KODA to large vision\-language datasets, we develop randomized low\-dimensional approximations of joint kernels using random projections, including Random Fourier Features for shift\-invariant kernels\. Empirically, KODA identifies consistent and interpretable discrepancy structures across vision\-language representations and provides sample subsets for representation alignment\. The code is available at[https://github\.com/yokiwuuu/KODA](https://github.com/yokiwuuu/KODA)\.

## 1Introduction

Multi\-modal embedding models have become a central component of modern machine learning systems, enabling joint representation of images, text, and other modalities within a shared semantic space\. Contrastive vision–language embeddings such as CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib1)\)and related paradigms including ALIGN\(Jiaet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib2)\), BLIP\(Liet al\.,[2022](https://arxiv.org/html/2606.04180#bib.bib3)\), and BLIP\-2\(Liet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib4)\)have demonstrated strong performance in cross\-modal retrieval and transfer, and they are now widely used as fixed representation interfaces in larger pipelines\. As a result, many vision–language foundation models and public variants coexist, differing in architecture, training objectives, and data sources\(Chertiet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib5)\)\. This diversity motivates principled methods for comparing representations and characterizing*how*they differ in the structure they induce on data, beyond reporting aggregate downstream metrics\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x1.png)

Figure 1:Overview of KODA for*Contrastive Embedding Clustering*, which aims to discover sample clusters that are represented differently by two embeddings\. We show KODA\-identified contrastive clusters for BLIP and CLIP embeddings on the MS\-COCO dataset\.Most existing comparisons of multi\-modal representations rely on downstream task performance, such as retrieval accuracy or zero\-shot classification on benchmark datasets\. While effective for ranking models, such evaluations provide limited insight into how representations organize data\. In vision–language settings, two models may achieve similar overall accuracy yet induce different groupings of image–text pairs, for example by emphasizing different semantic attributes, compositional patterns, or rare concepts\. Identifying such fine\-grained differences can support interpretability\-oriented workflows, including targeted data curation, model selection, and representation alignment\.

A recent line of work on interpretable embedding comparison has begun to study sample groups that are organized differently by two representations\. In particular, the SPEC framework byJalaliet al\.\([2025a](https://arxiv.org/html/2606.04180#bib.bib9)\)compares two embeddings by constructing kernel similarity matrices on a shared reference dataset and analyzing the eigendecomposition of their kernel\-difference matrix\. This setting motivates the task we call*Contrastive Embedding Clustering*: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another\. However, the kernel\-difference construction in SPEC does not explicitly enforce this asymmetric objective; its eigendirections may reflect structure present in both embeddings or aggregate several effects, rather than isolating subsets that are weakly grouped with respect to a specified reference embedding\.

In this work, we formulate Contrastive Embedding Clustering as a constrained optimization problem\. Given kernel similarity matricesKAK\_\{A\}andKBK\_\{B\}induced by two representations, KODA seeks components that are strongly grouped under one representation while explicitly constrained to be weakly grouped under the other\. We introduce*Kernel Optimization for Discrepancy Analysis*\(KODA\), which solves

maxx∈ℝn\\displaystyle\\max\_\{x\\in\\mathbb\{R\}^\{n\}\}\\qquadx⊤​KA​x\\displaystyle x^\{\\top\}K\_\{A\}\\,xsubject tox⊤​KB​x≤ϵ,\\displaystyle x^\{\\top\}K\_\{B\}\\,x\\leq\\epsilon,\(1\)‖x‖2=1\.\\displaystyle\\bigl\\\|x\\bigr\\\|\_\{2\}=1\.To identify multiple discrepancy modes, we solve a sequence of such problems with orthogonality constraints between the current and previous solutions\. Although this optimization problem is non\-convex, we show that it admits an efficient solution via structured spectral computations\.

We further extend KODA to the comparison of multi\-modal representations by constructing unified kernels through modality\-wise kernel multiplication\. This product\-kernel formulation enables direct comparison of representations over paired data, such as image–text samples, within the same constrained framework\. However, exact covariance\-operator implementations become infeasible in multi\-modal settings because effective feature dimensions can scale multiplicatively across modalities\.

To enable scalable computation, we develop approximations based on random projections and Random Fourier Features\(Rahimi and Recht,[2007](https://arxiv.org/html/2606.04180#bib.bib17)\), which reduce the effective dimensionality of the joint kernel feature map while preserving approximation guarantees\. We evaluate KODA on widely used vision–language representation models, including CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib1)\), ALIGN\(Jiaet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib2)\), and BLIP\-style models\(Liet al\.,[2022](https://arxiv.org/html/2606.04180#bib.bib3),[2023](https://arxiv.org/html/2606.04180#bib.bib4)\), using standard benchmarks such as MS\-COCO\(Linet al\.,[2014](https://arxiv.org/html/2606.04180#bib.bib12)\)\. Our experiments show that KODA identifies consistent discrepancy structures, improves over kernel\-difference baselines in finding multiple discrepancy modes, and provides sample subsets that can be used for representation alignment\.

## 2Related Work

Multi\-modal embedding models\.Vision–language embedding models are now a standard component in cross\-modal retrieval, zero\-shot recognition, and multi\-modal generative AI pipelines\. Representative approaches include CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib1)\), ALIGN\(Jiaet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib2)\), and BLIP / BLIP\-2\(Liet al\.,[2022](https://arxiv.org/html/2606.04180#bib.bib3),[2023](https://arxiv.org/html/2606.04180#bib.bib4)\), alongside scaling studies and public training efforts that broaden the space of available variants\(Chertiet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib5)\)\. SigLIP\(Zhaiet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib8)\)and SigLip 2\(Tschannenet al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib33)\)propose an alternative pre\-training objective based on a sigmoid loss\. While these models are commonly compared via downstream benchmarks, such evaluations often provide limited insight into how two embeddings organize the*same*paired data\.

Explainability in representation learning\.A broad line of research develops tools to interpret learned representations directly, often by linking internal directions or units to human\-understandable concepts\. Network Dissection\(Bauet al\.,[2017](https://arxiv.org/html/2606.04180#bib.bib28)\)quantifies interpretability of visual representations by measuring alignment between hidden units and semantic concepts\. Concept\-based explanation methods use examples of a concept to define directions in representation space: TCAV\(Kimet al\.,[2018](https://arxiv.org/html/2606.04180#bib.bib29)\)measures concept sensitivity via directional derivatives, while ACE\(Ghorbaniet al\.,[2019](https://arxiv.org/html/2606.04180#bib.bib30)\)automatically extracts concepts and evaluates their importance, reducing reliance on manually specified concept sets\. Concept Bottleneck Models\(Kohet al\.,[2020](https://arxiv.org/html/2606.04180#bib.bib31)\)further emphasize interpretability by explicitly structuring representations around a set of supervised concepts and enabling interventions on that representation\. More recently,Gonget al\.\([2025b](https://arxiv.org/html/2606.04180#bib.bib52)\)improve the visual interpretability of CLIP through unsupervised adversarial fine\-tuning with norm regularization, showing gains through feature\-attribution and network\-dissection analyses\. These works motivate analyses that identify*which parts of a dataset*correspond to salient representational structure, but they do not directly target the embeddings’ discrepancy discovery\.

Kernel\-based methods for feature representations\.Kernel methods provide a flexible way to compare, align, fuse, and evaluate learned feature representations through pairwise similarity structure rather than only downstream task accuracy\. Recent work has used kernel matrices to explain differences between embedding spaces and align their induced cluster structures\(Jalaliet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib9)\), and to improve vision\-language representations by aligning CLIP visual embeddings with stronger vision\-centric embeddings such as DINOv2\(Gonget al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib35)\)\. Complementarily, kernel product feature maps and maximum kernel entropy methods have been used to fuse and recover embeddings while preserving pairwise similarity information\(Wuet al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib37); Wu and Farnia,[2026](https://arxiv.org/html/2606.04180#bib.bib36)\)\. Kernel\-based scores have also become central to generative\-model evaluation, beginning with MMD\-based comparison\(Grettonet al\.,[2012](https://arxiv.org/html/2606.04180#bib.bib38)\)and the Kernel Inception Distance\(Bińkowskiet al\.,[2018](https://arxiv.org/html/2606.04180#bib.bib39); Wanget al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib32)\), and extending to entropy\- and spectrum\-based measures for diversity, novelty, and prompt\-aware evaluation, including RKE\(Jalaliet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib40)\), Vendi\(Friedman and Dieng,[2023](https://arxiv.org/html/2606.04180#bib.bib41); Ospanovet al\.,[2024](https://arxiv.org/html/2606.04180#bib.bib43)\), KEN\(Zhanget al\.,[2024](https://arxiv.org/html/2606.04180#bib.bib42),[2025](https://arxiv.org/html/2606.04180#bib.bib44)\), and conditional kernel entropy\(Jalaliet al\.,[2025b](https://arxiv.org/html/2606.04180#bib.bib45); Ospanovet al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib46); Jalaliet al\.,[2026](https://arxiv.org/html/2606.04180#bib.bib47)\)\. Kernel\-based uses of embeddings have also been explored for model evaluation, selection, and mixture construction\(Steinet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib49); Huet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib53); Rezaeiet al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib48); Huet al\.,[2025b](https://arxiv.org/html/2606.04180#bib.bib50); Jafari and Farnia,[2026](https://arxiv.org/html/2606.04180#bib.bib51)\)\. These methods motivate viewing embeddings and foundation models not only as feature vectors for prediction, but also as kernel\-induced representations whose geometry can be systematically compared and evaluated\.

## 3Preliminaries

### 3\.1Embedding maps & comparison setting

Let𝒳\\mathcal\{X\}denote an input space and letψ:𝒳→𝒮\\psi:\\mathcal\{X\}\\to\\mathcal\{S\}be an embedding map into a representation space𝒮\\mathcal\{S\}\(typically Euclidean\)\. We consider two embedding maps

ψ1:𝒳→𝒮1,ψ2:𝒳→𝒮2,\\psi\_\{1\}:\\mathcal\{X\}\\to\\mathcal\{S\}\_\{1\},\\qquad\\psi\_\{2\}:\\mathcal\{X\}\\to\\mathcal\{S\}\_\{2\},which may have different output dimensions and thus induce different similarity structures on the same inputs\. We assume access to a reference dataset\{xi\}i=1n⊂𝒳\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}\\subset\\mathcal\{X\}sampled from an underlying distribution\. Our goal is to compareψ1\\psi\_\{1\}andψ2\\psi\_\{2\}through the geometry they induce on this reference set, without relying on labeled downstream tasks\.

### 3\.2Kernel functions & kernel\-induced quadratic forms

A kernel functionk:𝒳×𝒳→ℝk:\\mathcal\{X\}\\times\\mathcal\{X\}\\to\\mathbb\{R\}assigns a similarity score and admits a feature mapϕ:𝒳→ℋ\\phi:\\mathcal\{X\}\\to\\mathcal\{H\}into a \(possibly infinite\-dimensional\) Hilbert spaceℋ\\mathcal\{H\}such that

k​\(x,x′\)=⟨ϕ​\(x\),ϕ​\(x′\)⟩ℋ\.k\(x,x^\{\\prime\}\)=\\langle\\phi\(x\),\\phi\(x^\{\\prime\}\)\\rangle\_\{\\mathcal\{H\}\}\.Given samplesx1,…,xnx\_\{1\},\\ldots,x\_\{n\}, the associated kernel matrixK∈ℝn×nK\\in\\mathbb\{R\}^\{n\\times n\}is

Ki​j=k​\(xi,xj\),K\_\{ij\}=k\(x\_\{i\},x\_\{j\}\),\(2\)and is positive semidefinite\. Common normalized examples include the cosine kernelkcos​\(u,v\)=u⊤​v‖u‖2​‖v‖2k\_\{\\mathrm\{cos\}\}\(u,v\)=\\frac\{u^\{\\top\}v\}\{\\\|u\\\|\_\{2\}\\\|v\\\|\_\{2\}\}\(for nonzerou,vu,v\) and the Gaussian \(RBF\) kernelkrbf​\(u,v\)=exp⁡\(−‖u−v‖22/\(2​σ2\)\)k\_\{\\mathrm\{rbf\}\}\(u,v\)=\\exp\\\!\\big\(\-\\\|u\-v\\\|\_\{2\}^\{2\}/\(2\\sigma^\{2\}\)\\big\), both satisfyingk​\(x,x\)=1k\(x,x\)=1\.

For anyv∈ℝnv\\in\\mathbb\{R\}^\{n\}with‖v‖2=1\\\|v\\\|\_\{2\}=1, the quadratic form

is the Rayleigh quotient ofKKatvv, and hence lies in\[λmin​\(K\),λmax​\(K\)\]\[\\lambda\_\{\\min\}\(K\),\\lambda\_\{\\max\}\(K\)\]and is maximized by a top eigenvector ofKK\. Moreover, whenϕ\\phiis finite\-dimensional andK=Φ​Φ⊤K=\\Phi\\Phi^\{\\top\}withΦi:=ϕ​\(xi\)⊤\\Phi\_\{i:\}=\\phi\(x\_\{i\}\)^\{\\top\}, we have the identity

v⊤​K​v=‖Φ⊤​v‖22=‖∑i=1nvi​ϕ​\(xi\)‖22,v^\{\\top\}Kv\\;=\\;\\\|\\Phi^\{\\top\}v\\\|\_\{2\}^\{2\}\\;=\\;\\Big\\\|\\sum\_\{i=1\}^\{n\}v\_\{i\}\\,\\phi\(x\_\{i\}\)\\Big\\\|\_\{2\}^\{2\},\(4\)which will be useful when interpreting and optimizing kernel\-based criteria on the reference set\. Whenϕ​\(x\)∈ℝd\\phi\(x\)\\in\\mathbb\{R\}^\{d\}, the empirical covariance matrix \(operator\) is

CX:=1n​Φ⊤​Φ=1n​∑i=1nϕ​\(xi\)​ϕ​\(xi\)⊤∈ℝd×d\.C\_\{X\}:=\\frac\{1\}\{n\}\\Phi^\{\\top\}\\Phi=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\phi\(x\_\{i\}\)\\phi\(x\_\{i\}\)^\{\\top\}\\in\\mathbb\{R\}^\{d\\times d\}\.\(5\)The matricesK/nK/nandCXC\_\{X\}share the same non\-zero eigenvalues \(including multiplicities\), since they are products ofΦ\\PhiandΦ⊤\\Phi^\{\\top\}in opposite orders\.

### 3\.3Shift\-invariant kernels & random Fourier features

To scale kernel computations, we use random Fourier features \(RFF\)\(Rahimi and Recht,[2007](https://arxiv.org/html/2606.04180#bib.bib17); Sutherland and Schneider,[2015](https://arxiv.org/html/2606.04180#bib.bib34)\)for shift\-invariant kernels onℝd\\mathbb\{R\}^\{d\}\. Consider kernels of the formk​\(x,x′\)=κ​\(x−x′\)k\(x,x^\{\\prime\}\)=\\kappa\(x\-x^\{\\prime\}\), whereκ\\kappais continuous and positive definite\. By Bochner’s theorem, there exists a non\-negative finite measure \(taking non\-negative values\)κ^\\widehat\{\\kappa\}, which is the Fourier transform ofκ\\kappa, such that

κ​\(δ\)=∫ℝdexp⁡\(i​ω⊤​δ\)​κ^​\(ω\)​dω\.\\kappa\(\\delta\)=\\int\_\{\\mathbb\{R\}^\{d\}\}\\exp\\bigl\(\\mathrm\{i\}\\omega^\{\\top\}\\delta\\bigr\)\\widehat\{\\kappa\}\(\\omega\)\\mathrm\{d\}\\omega\.\(6\)For a normalized kernel withκ​\(0\)=1\\kappa\(0\)=1,κ^\\widehat\{\\kappa\}will be a probability measure\. We then sampleω1,…,ωm∼i\.i\.d\.κ^\\omega\_\{1\},\\ldots,\\omega\_\{m\}\\stackrel\{\{\\scriptstyle\\mathrm\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\widehat\{\\kappa\}and define the RFF proxy feature mapφr:𝒳→ℝ2​r\\varphi\_\{r\}:\\mathcal\{X\}\\rightarrow\\mathbb\{R\}^\{2r\}as

φr\(x\)=1r\[cos\(ω1⊤x\),sin\(ω1⊤x\),\.,cos\(ωr⊤x\),sin\(ωr⊤x\)\]\.\\varphi\_\{r\}\(x\)=\\frac\{1\}\{\\sqrt\{r\}\}\\bigl\[\\cos\(\\omega\_\{1\}^\{\\top\}x\),\\sin\(\\omega\_\{1\}^\{\\top\}x\),\.,\\cos\(\\omega\_\{r\}^\{\\top\}x\),\\sin\(\\omega\_\{r\}^\{\\top\}x\)\\bigr\]\.\(7\)A direct calculation yields𝔼​\[⟨φr​\(x\),φr​\(x′\)⟩\]=k​\(x,x′\)\\mathbb\{E\}\\\!\\left\[\\langle\\varphi\_\{r\}\(x\),\\varphi\_\{r\}\(x^\{\\prime\}\)\\rangle\\right\]=k\(x,x^\{\\prime\}\), where the expectation is over the sampled frequency vectors\. Given\{xi\}i=1n\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}, we form the approximate kernel matrixK~\\widetilde\{K\}byK~i​j=⟨φr​\(xi\),φr​\(xj\)⟩\\widetilde\{K\}\_\{ij\}=\\langle\\varphi\_\{r\}\(x\_\{i\}\),\\varphi\_\{r\}\(x\_\{j\}\)\\rangle, enabling computation of kernel quadratic forms and spectral quantities using an explicit feature representation\.

## 4KODA: Optimization\-based discrepancy identification in kernel matrices

We formalize the comparison problem as a*Contrastive Embedding Clustering*task\. Given two embeddingsAAandBBevaluated on the same reference set\{x1,…,xn\}\\\{x\_\{1\},\\ldots,x\_\{n\}\\\}, the goal is to identify subsets or signed directions over the reference samples that form a coherent cluster under one embedding but not under the other\. In this sense, the task is not merely to measure a global discrepancy between embeddings, but to localize it by extracting directions in the reference set where the two kernel similarity geometries disagree\.

For two embeddingsAAandBB, we are given their normalized kernel similarity matricesKA,KB∈ℝn×nK\_\{A\},K\_\{B\}\\in\\mathbb\{R\}^\{n\\times n\}constructed on the same reference set\{x1,…,xn\}\\\{x\_\{1\},\\ldots,x\_\{n\}\\\}, where1n​KA⪰0\\frac\{1\}\{n\}K\_\{A\}\\succeq 0and1n​KB⪰0\\frac\{1\}\{n\}K\_\{B\}\\succeq 0are PSD and unit\-trace\. We develop an optimization formulation for Contrastive Embedding Clustering: extracting directions on the reference set along which the structure induced by one embedding is strongly clustered while the structure by the other is weakly clustered\.

### 4\.1Constrained quadratic programming for kernel\-based embedding comparison

For a target levelϵ\>0\\epsilon\>0, we consider the following quadratically constrained program for Contrastive Embedding Clustering\. The constraint enforces weak clusterability underKBK\_\{B\}, while the objective searches for a direction that is maximally clustered underKAK\_\{A\}:

maxx∈ℝnx⊤​KA​xs\.t\.x⊤​KB​x≤ϵ,‖x‖22=1\.\\displaystyle\\begin\{aligned\} \\max\_\{x\\in\\mathbb\{R\}^\{n\}\}\\quad&x^\{\\top\}K\_\{A\}\\,x\\\\ \\text\{s\.t\.\}\\quad&x^\{\\top\}K\_\{B\}\\,x\\leq\\epsilon,\\\\ &\\bigl\\\|x\\bigr\\\|^\{2\}\_\{2\}=1\.\\end\{aligned\}\(8\)Letx⋆x^\{\\star\}denote an optimizer\. The constraintx⊤​KB,x≤ϵx^\{\\top\}K\_\{B\},x\\leq\\epsilonenforces that the discrepancy direction has limited energy under the second embedding, while the objective selects the direction with the strongest similarity concentration under the first embedding\. Thus,x⋆x^\{\\star\}identifies a contrastive cluster direction: a set\-level pattern that is salient in embeddingAAbut suppressed, diffuse, or absent in embeddingBB\.

Although the optimization problem \([8](https://arxiv.org/html/2606.04180#S4.E8)\) is a non\-convex optimization task \(maximizing a convex objective function\), its optimizers admit an eigenvector\-based characterization as revealed by KKT conditions\.

###### Proposition 4\.1\(Eigenvector form of an optimizer\)\.

Assume \([8](https://arxiv.org/html/2606.04180#S4.E8)\) is feasible and there existsx¯\\bar\{x\}with‖x¯‖2=1\\\|\\bar\{x\}\\\|\_\{2\}=1andx¯⊤​KB​x¯<ϵ\\bar\{x\}^\{\\top\}K\_\{B\}\\,\\bar\{x\}<\\epsilon\. Then, there exist scalarsλ⋆≥0\\lambda^\{\\star\}\\geq 0andν⋆∈ℝ\\nu^\{\\star\}\\in\\mathbb\{R\}such that any optimizerx⋆x^\{\\star\}with unit\-norm \(‖x⋆‖2=1\\\|x^\{\\star\}\\\|\_\{2\}=1\) satisfies

\(KA−λ⋆​KB\)​x⋆\\displaystyle\(K\_\{A\}\\,\-\\lambda^\{\\star\}K\_\{B\}\\,\)\\,x^\{\\star\}=ν⋆​x⋆,\\displaystyle=\\nu^\{\\star\}x^\{\\star\},\(9\)λ⋆​\(x⋆⊤​KB​x⋆−ϵ\)\\displaystyle\\lambda^\{\\star\}\\big\(\{x^\{\\star\}\}^\{\\top\}K\_\{B\}\\,x^\{\\star\}\-\\epsilon\\big\)=0\.\\displaystyle=0\.\(10\)

###### Proof\.

We present the proof in the Appendix\. ∎

#### Searching overλ\\lambda\.

Proposition[4\.1](https://arxiv.org/html/2606.04180#S4.Thmtheorem1)motivates a one\-dimensional search overλ≥0\\lambda\\geq 0: for eachλ\\lambda, compute a leading eigenvectorxλx\_\{\\lambda\}ofKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,\(normalized to‖xλ‖2=1\\\|x\_\{\\lambda\}\\\|\_\{2\}=1\) and evaluateg​\(λ\):=xλ⊤​KB​xλg\(\\lambda\):=x\_\{\\lambda\}^\{\\top\}K\_\{B\}\\,x\_\{\\lambda\}\. We then select aλ\\lambdathat yieldsg​\(λ\)≤ϵg\(\\lambda\)\\leq\\epsilonand maximizesxλ⊤​KA​xλx\_\{\\lambda\}^\{\\top\}K\_\{A\}\\,x\_\{\\lambda\}among such candidates \(up to numerical tolerance\)\.

### 4\.2Iterative extraction of discrepancy directions via KODA

A single solution of \([8](https://arxiv.org/html/2606.04180#S4.E8)\) yields one contrastive cluster direction\. KODA solves the Contrastive Embedding Clustering task by extracting such directions, iteratively solving \([8](https://arxiv.org/html/2606.04180#S4.E8)\) while enforcing orthogonality to the found directions\.

Letx1,…,xt−1x\_\{1\},\\ldots,x\_\{t\-1\}be previously extracted unit vectors, and letUt−1∈ℝn×\(t−1\)U\_\{t\-1\}\\in\\mathbb\{R\}^\{n\\times\(t\-1\)\}have orthonormal columns spanningspan​\{x1,…,xt−1\}\\mathrm\{span\}\\\{x\_\{1\},\\ldots,x\_\{t\-1\}\\\}\. Define the orthogonal projector

Pt−1:=I−Ut−1​Ut−1⊤\.\\displaystyle P\_\{t\-1\}\\;:=\\;I\-U\_\{t\-1\}U\_\{t\-1\}^\{\\top\}\.\(11\)At iterationtt, we solve

maxx∈ℝnx⊤​KA​xs\.t\.x⊤​KB​x≤ϵ,‖x‖2=1,Ut−1⊤​x=0\.\\displaystyle\\begin\{aligned\} \\max\_\{x\\in\\mathbb\{R\}^\{n\}\}\\quad&x^\{\\top\}K\_\{A\}\\,x\\\\ \\text\{s\.t\.\}\\quad&x^\{\\top\}K\_\{B\}\\,x\\leq\\epsilon,\\\\ &\\\|x\\\|\_\{2\}=1,\\\\ &U\_\{t\-1\}^\{\\top\}x=0\.\\end\{aligned\}\(12\)
###### Proposition 4\.2\.

The optimization problem \([12](https://arxiv.org/html/2606.04180#S4.E12)\) is equivalent to

maxx∈ℝnx⊤​\(Pt−1​KA​Pt−1\)​xs\.t\.x⊤​\(Pt−1​KB​Pt−1\)​x≤ϵ,‖x‖2=1,\\displaystyle\\begin\{aligned\} \\max\_\{x\\in\\mathbb\{R\}^\{n\}\}\\quad&x^\{\\top\}\(P\_\{t\-1\}K\_\{A\}\\,P\_\{t\-1\}\)x\\\\ \\text\{s\.t\.\}\\quad&x^\{\\top\}\(P\_\{t\-1\}K\_\{B\}\\,P\_\{t\-1\}\)x\\leq\\epsilon,\\\\ &\\\|x\\\|\_\{2\}=1,\\end\{aligned\}\(13\)and every optimizer of \([13](https://arxiv.org/html/2606.04180#S4.E13)\) satisfiesUt−1⊤​x=0U\_\{t\-1\}^\{\\top\}x=0\.

###### Proof\.

We present the proof in the Appendix\. ∎

Algorithm[1](https://arxiv.org/html/2606.04180#alg1)summarizes the steps in KODA\. At each iteration, we work with the projected matricesA=P​KA​PA=PK\_\{A\}\\,PandB=P​KB​PB=PK\_\{B\}\\,Pfrom \([13](https://arxiv.org/html/2606.04180#S4.E13)\) and perform a 1D search overλ\\lambdavia repeated leading\-eigenvector computations\.

### 4\.3Scalable principal\-eigenvector computation via covariance blocks

KODA requires repeated computation of a principal eigenvector of matrices of the formKA−λ​KB∈ℝn×nK\_\{A\}\\,\-\\lambda K\_\{B\}\\,\\in\\mathbb\{R\}^\{n\\times n\}\. For large reference sets \(e\.g\.,n⪆n\\gtrapprox\\,20000\), this step can be a computational bottleneck in the dense\-kernel regime: even iterative eigensolvers \(Lanczos/power iteration\) rely on repeated matrix–vector products, each costingΘ​\(n2\)\\Theta\(n^\{2\}\)time \(andΘ​\(n2\)\\Theta\(n^\{2\}\)memory ifKA,KBK\_\{A\}\\,,K\_\{B\}\\,are explicitly formed\)\. When the kernels admit explicit feature representations with feature dimensionsd1,d2≪nd\_\{1\},d\_\{2\}\\ll n\(e\.g\., via random features\), the same principal\-eigenvector computation can be reduced to an eigenproblem of dimensiond1\+d2d\_\{1\}\+d\_\{2\}\.

Assume the kernel matrices factor as

KA\\displaystyle K\_\{A\}\\,=ΦA​ΦA⊤,KB=ΦB​ΦB⊤,\\displaystyle=\\Phi\_\{A\}\\Phi\_\{A\}^\{\\top\},\\qquad K\_\{B\}\\,=\\Phi\_\{B\}\\Phi\_\{B\}^\{\\top\},\(14\)withΦA∈ℝn×d1\\Phi\_\{A\}\\in\\mathbb\{R\}^\{n\\times d\_\{1\}\}andΦB∈ℝn×d2\\Phi\_\{B\}\\in\\mathbb\{R\}^\{n\\times d\_\{2\}\}\. LetΦ:=\[ΦA​ΦB\]∈ℝn×\(d1\+d2\)\\Phi:=\[\\Phi\_\{A\}\\;\\;\\Phi\_\{B\}\]\\in\\mathbb\{R\}^\{n\\times\(d\_\{1\}\+d\_\{2\}\)\}and define the covariance blocks

CA​A\\displaystyle C\_\{AA\}:=ΦA⊤​ΦA,CA​B:=ΦA⊤​ΦB,\\displaystyle:=\\Phi\_\{A\}^\{\\top\}\\Phi\_\{A\},\\qquad C\_\{AB\}:=\\Phi\_\{A\}^\{\\top\}\\Phi\_\{B\},CB​A\\displaystyle C\_\{BA\}:=ΦB⊤​ΦA,CB​B:=ΦB⊤​ΦB,\\displaystyle:=\\Phi\_\{B\}^\{\\top\}\\Phi\_\{A\},\\qquad C\_\{BB\}:=\\Phi\_\{B\}^\{\\top\}\\Phi\_\{B\},\(15\)so thatG:=Φ⊤​Φ=\[CA​ACA​BCB​ACB​B\]G:=\\Phi^\{\\top\}\\Phi=\\begin\{bmatrix\}C\_\{AA\}&C\_\{AB\}\\vskip 2\.84526pt\\\\ C\_\{BA\}&C\_\{BB\}\\end\{bmatrix\}\.

###### Proposition 4\.3\.

For coefficientλ≥0\\lambda\\geq 0, define block matricesSλ:=diag​\(Id1,−λ​Id2\)S\_\{\\lambda\}:=\\mathrm\{diag\}\(I\_\{d\_\{1\}\},\-\\lambda I\_\{d\_\{2\}\}\)and

Mλ:=Sλ​G=\[CA​ACA​B−λ​CB​A−λ​CB​B\]\.\\displaystyle M\_\{\\lambda\}\\;:=\\;S\_\{\\lambda\}G\\;=\\;\\begin\{bmatrix\}C\_\{AA\}&C\_\{AB\}\\vskip 2\.84526pt\\\\ \-\\lambda C\_\{BA\}&\-\\lambda C\_\{BB\}\\end\{bmatrix\}\.\(16\)Letηλ:=λmax​\(KA−λ​KB\)\\eta\_\{\\lambda\}:=\\lambda\_\{\\max\}\(K\_\{A\}\\,\-\\lambda K\_\{B\}\\,\)and letuλu\_\{\\lambda\}be a \(right\) eigenvector ofMλM\_\{\\lambda\}associated with eigenvalueηλ\\eta\_\{\\lambda\}\. Then,xλ:=Φ​uλx\_\{\\lambda\}:=\\Phi u\_\{\\lambda\}is a principal eigenvector ofKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,\(after normalization\)\.

###### Proof\.

We present the proof in the Appendix\. ∎

#### Discussion and computational implications\.

Proposition[4\.3](https://arxiv.org/html/2606.04180#S4.Thmtheorem3)reduces the principal\-eigenvector computation of then×nn\\times nmatrixKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,to an eigenproblem of size\(d1\+d2\)×\(d1\+d2\)\(d\_\{1\}\+d\_\{2\}\)\\times\(d\_\{1\}\+d\_\{2\}\), which is independent ofnn\. This is particularly beneficial whend1,d2≪nd\_\{1\},d\_\{2\}\\ll n, as is typical with explicit feature maps or random features\. In practice, we form the covariance blocks in \([4\.3](https://arxiv.org/html/2606.04180#S4.Ex5)\) in𝒪​\(n​\(d1\+d2\)2\)\\mathcal\{O\}\(n\(d\_\{1\}\+d\_\{2\}\)^\{2\}\)time \(or𝒪​\(n​\(d1\+d2\)\)\\mathcal\{O\}\(n\(d\_\{1\}\+d\_\{2\}\)\)time ifCi​jC\_\{ij\}are accumulated online with a single pass\) and then compute a dominant eigenvector ofMλM\_\{\\lambda\}\. The lifted vectorxλ=Φ​uλx\_\{\\lambda\}=\\Phi u\_\{\\lambda\}can be obtained without forming anyn×nn\\times nmatrix, and can be normalized to satisfy‖xλ‖2=1\\\|x\_\{\\lambda\}\\\|\_\{2\}=1before evaluating the constraint quantityxλ⊤​KB​xλx\_\{\\lambda\}^\{\\top\}K\_\{B\}\\,x\_\{\\lambda\}\(which can likewise be computed viaΦB\\Phi\_\{B\}asxλ⊤​KB​xλ=‖ΦB⊤​xλ‖22x\_\{\\lambda\}^\{\\top\}K\_\{B\}\\,x\_\{\\lambda\}=\\\|\\Phi\_\{B\}^\{\\top\}x\_\{\\lambda\}\\\|\_\{2\}^\{2\}\)\.

#### Sample complexity of the covariance\-block eigendirections\.

To state a population sample\-complexity guarantee, we use the*normalized*covariance blocksC^i​j:=1n​Φi⊤​Φj\\widehat\{C\}\_\{ij\}:=\\frac\{1\}\{n\}\\Phi\_\{i\}^\{\\top\}\\Phi\_\{j\}andG^:=1n​Φ⊤​Φ\\widehat\{G\}:=\\frac\{1\}\{n\}\\Phi^\{\\top\}\\Phi\. Letz​\(x\):=\[ΦA​\(x\);ΦB​\(x\)\]∈ℝd1\+d2z\(x\):=\[\\Phi\_\{A\}\(x\);\\Phi\_\{B\}\(x\)\]\\in\\mathbb\{R\}^\{d\_\{1\}\+d\_\{2\}\}denote the per\-sample feature vector \(theiith row ofΦ\\Phiisz​\(xi\)⊤z\(x\_\{i\}\)^\{\\top\}\)\. Assume‖ΦA​\(x\)‖2≤1\\\|\\Phi\_\{A\}\(x\)\\\|\_\{2\}\\leq 1and‖ΦB​\(x\)‖2≤1\\\|\\Phi\_\{B\}\(x\)\\\|\_\{2\}\\leq 1for allxx\(equivalently,‖z​\(x\)‖22≤2\\\|z\(x\)\\\|\_\{2\}^\{2\}\\leq 2\)\. Define the population block covarianceG:=𝔼​\[z​\(X\)​z​\(X\)⊤\]G:=\\mathbb\{E\}\[z\(X\)z\(X\)^\{\\top\}\]andSλ:=diag​\(Id1,−λ​Id2\)S\_\{\\lambda\}:=\\mathrm\{diag\}\(I\_\{d\_\{1\}\},\-\\lambda I\_\{d\_\{2\}\}\)\. Finally, define the symmetric matricesBλ:=G1/2​Sλ​G1/2B\_\{\\lambda\}:=G^\{1/2\}S\_\{\\lambda\}G^\{1/2\}andB^λ:=G^1/2​Sλ​G^1/2\\widehat\{B\}\_\{\\lambda\}:=\\widehat\{G\}^\{1/2\}S\_\{\\lambda\}\\widehat\{G\}^\{1/2\}\.

###### Theorem 4\.4\.

Consider the setting described above\. Then, for everyλ≥0\\lambda\\geq 0andδ∈\(0,1\)\\delta\\in\(0,1\), the following holds with probability at least1−δ1\-\\delta,

‖B^λ−Bλ‖2≤12​‖Sλ‖2​d1\+d2n4​\(1\+log⁡\(1/δ\)\)\.\\displaystyle\\bigl\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\bigr\\\|\_\{2\}\\\>\\leq\\\>12\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,\\sqrt\[4\]\{\\frac\{d\_\{1\}\+d\_\{2\}\}\{n\}\}\\,\\Bigl\(1\+\\sqrt\{\\log\(1/\\delta\)\}\\Bigr\)\.Moreover, ifBλB\_\{\\lambda\}has eigengapγλ:=λ1​\(Bλ\)−λ2​\(Bλ\)\>0\\gamma\_\{\\lambda\}:=\\lambda\_\{1\}\(B\_\{\\lambda\}\)\-\\lambda\_\{2\}\(B\_\{\\lambda\}\)\>0, andv1,v^1v\_\{1\},\\widehat\{v\}\_\{1\}are unit top eigenvectors ofBλ,B^λB\_\{\\lambda\},\\widehat\{B\}\_\{\\lambda\}, then

sin⁡∠​\(v^1,v1\)≤‖B^λ−Bλ‖2γλ\.\\displaystyle\\sin\\angle\(\\widehat\{v\}\_\{1\},v\_\{1\}\)\\;\\leq\\;\\frac\{\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{2\}\}\{\\gamma\_\{\\lambda\}\}\.\(17\)

###### Proof\.

We present the proof in the Appendix\. ∎

Algorithm 1KODA: Kernel Optimization for Discrepancy Analysis0:PSD kernels

KA,KB∈ℝn×nK\_\{A\}\\,,K\_\{B\}\\,\\in\\mathbb\{R\}^\{n\\times n\}, threshold

ϵ\>0\\epsilon\>0, number of directions

TT, tolerance

τ\\tau
0:Directions

x1,…,xT∈ℝnx\_\{1\},\\ldots,x\_\{T\}\\in\\mathbb\{R\}^\{n\}
1:

U←\[\]U\\leftarrow\[\\ \]\(null matrix\)

2:for

t=1,…,Tt=1,\\ldots,Tdo

3:

P←I−U​U⊤P\\leftarrow I\-UU^\{\\top\}
4:

A←P​KA​PA\\leftarrow PK\_\{A\}\\,P,

B←P​KB​PB\\leftarrow PK\_\{B\}\\,P
5:Choose

λt≥0\\lambda\_\{t\}\\geq 0by a 1D search using: compute a leading eigenvector

xλx\_\{\\lambda\}of

A−λ​BA\-\\lambda Bwith

‖xλ‖2=1\\\|x\_\{\\lambda\}\\\|\_\{2\}=1, and evaluate

g​\(λ\)=xλ⊤​B​xλg\(\\lambda\)=x\_\{\\lambda\}^\{\\top\}Bx\_\{\\lambda\}
6:Stop when

g​\(λt\)≤ϵ\+τg\(\\lambda\_\{t\}\)\\leq\\epsilon\+\\tauand set

xt←xλtx\_\{t\}\\leftarrow x\_\{\\lambda\_\{t\}\}
7:Orthonormalize:

U←orth​\(\[U​xt\]\)U\\leftarrow\\mathrm\{orth\}\(\[U\\;\\;x\_\{t\}\]\)
8:endfor

9:Return

\{xt\}t=1T\\\{x\_\{t\}\\\}\_\{t=1\}^\{T\}

## 5KODA for Multi\-modal embeddings via product kernels and random features

We extend KODA to the comparison of multi\-modal embeddings, where each reference item consists of paired observations from different modalities\. We first introduce a product\-kernel formulation that induces a joint similarity matrix on paired samples\. We then address the computational challenge posed by the tensor\-product feature space of product kernels by proposing a joint random Fourier feature approximation, and establish a guarantee on the stability of the leading eigenspaces used by KODA\.

### 5\.1Product\-kernel formulation and tensor\-product bottleneck

For each reference sample that paired iszi=\(xi,ti\)z\_\{i\}=\(x\_\{i\},t\_\{i\}\), let

ui=ψx​\(xi\)∈ℝdx,vi=ψt​\(ti\)∈ℝdt\.u\_\{i\}=\\psi\_\{x\}\(x\_\{i\}\)\\in\\mathbb\{R\}^\{d\_\{x\}\},\\qquad v\_\{i\}=\\psi\_\{t\}\(t\_\{i\}\)\\in\\mathbb\{R\}^\{d\_\{t\}\}\.We consider normalized shift\-invariant kernels for each modality as follows whereκx​\(0\)=κt​\(0\)=1\\kappa\_\{x\}\(0\)=\\kappa\_\{t\}\(0\)=1:

kx​\(u,u′\)=κx​\(u−u′\),kt​\(v,v′\)=κt​\(v−v′\)\\displaystyle k\_\{x\}\(u,u^\{\\prime\}\)=\\kappa\_\{x\}\(u\-u^\{\\prime\}\),\\;\\;k\_\{t\}\(v,v^\{\\prime\}\)=\\kappa\_\{t\}\(v\-v^\{\\prime\}\)\(18\)Then, we define the multi\-modal product kernel as:

k​\(\(u,v\),\(u′,v′\)\)\\displaystyle k\\big\(\(u,v\),\(u^\{\\prime\},v^\{\\prime\}\)\\big\)=kx​\(u,u′\)​kt​\(v,v′\)\\displaystyle=k\_\{x\}\(u,u^\{\\prime\}\)\\,k\_\{t\}\(v,v^\{\\prime\}\)=κx​\(u−u′\)​κt​\(v−v′\)\.\\displaystyle=\\kappa\_\{x\}\(u\-u^\{\\prime\}\)\\,\\kappa\_\{t\}\(v\-v^\{\\prime\}\)\.\(19\)For a reference set\{\(ui,vi\)\}i=1n\\\{\(u\_\{i\},v\_\{i\}\)\\\}\_\{i=1\}^\{n\}, the corresponding kernel matrix satisfiesK=Kx⊙KtK=K\_\{x\}\\odot K\_\{t\}where

\(Kx\)i​j=kx​\(ui,uj\),\(Kt\)i​j=kt​\(vi,vj\),\\displaystyle\(K\_\{x\}\)\_\{ij\}=k\_\{x\}\(u\_\{i\},u\_\{j\}\),\\quad\(K\_\{t\}\)\_\{ij\}=k\_\{t\}\(v\_\{i\},v\_\{j\}\),\(20\)with⊙\\odotdenoting the Hadamard product\. At the kernel level, KODA applies directly by replacing unimodal kernels in Section[4](https://arxiv.org/html/2606.04180#S4)withKK\.

The challenge arises in covariance\-based implementations: the feature map associated with \([5\.1](https://arxiv.org/html/2606.04180#S5.Ex8)\) is the tensor product

ϕ​\(u,v\)=ϕx​\(u\)⊗ϕt​\(v\),\\phi\(u,v\)=\\phi\_\{x\}\(u\)\\otimes\\phi\_\{t\}\(v\),whose ambient dimension scales asdx​dtd\_\{x\}d\_\{t\}\. This renders covariance\-space eigen\-computations infeasible for standard multi\-modal embeddings, motivating a low\-dimensional kernel approximation\.

### 5\.2Joint random Fourier features

Sinceκx\\kappa\_\{x\}andκt\\kappa\_\{t\}are continuous, real\-valued, and shift\-invariant, Bochner’s theorem yields probability measuresκ^x\\widehat\{\\kappa\}\_\{x\}andκ^t\\widehat\{\\kappa\}\_\{t\}such that

κx​\(δ\)\\displaystyle\\kappa\_\{x\}\(\\delta\)=∫ℝdxκ^x​\(ωx\)​cos⁡\(ωx⊤​δ\)​dωx,\\displaystyle=\\int\_\{\\mathbb\{R\}^\{d\_\{x\}\}\}\\widehat\{\\kappa\}\_\{x\}\(\\omega\_\{x\}\)\\cos\(\\omega\_\{x\}^\{\\top\}\\delta\)\\,\\mathrm\{d\}\\omega\_\{x\},κt​\(ζ\)\\displaystyle\\kappa\_\{t\}\(\\zeta\)=∫ℝdtκ^t​\(ωt\)​cos⁡\(ωt⊤​ζ\)​dωt\.\\displaystyle=\\int\_\{\\mathbb\{R\}^\{d\_\{t\}\}\}\\widehat\{\\kappa\}\_\{t\}\(\\omega\_\{t\}\)\\cos\(\\omega\_\{t\}^\{\\top\}\\zeta\)\\,\\mathrm\{d\}\\omega\_\{t\}\.\(21\)We samplerrfrequency vectors independently for each modality:

ωx,1,…,ωx,r∼iidκ^x,ωt,1,…,ωt,r∼iidκ^t,\\omega\_\{x,1\},\\ldots,\\omega\_\{x,r\}\\stackrel\{\{\\scriptstyle\\text\{\\rm iid\}\}\}\{\{\\sim\}\}\\widehat\{\\kappa\}\_\{x\},\\quad\\omega\_\{t,1\},\\ldots,\\omega\_\{t,r\}\\stackrel\{\{\\scriptstyle\\text\{\\rm iid\}\}\}\{\{\\sim\}\}\\widehat\{\\kappa\}\_\{t\},and define the joint random Fourier feature map

φ\(u,v\)=1r\[\\displaystyle\\varphi\(u,v\)=\\frac\{1\}\{\\sqrt\{r\}\}\\Bigl\[cos⁡\(ωx,1⊤​u\+ωt,1⊤​v\),sin⁡\(ωx,1⊤​u\+ωt,1⊤​v\),\\displaystyle\\cos\(\\omega\_\{x,1\}^\{\\top\}u\+\\omega\_\{t,1\}^\{\\top\}v\),\\sin\(\\omega\_\{x,1\}^\{\\top\}u\+\\omega\_\{t,1\}^\{\\top\}v\),…,cos\(ωx,r⊤u\+\\displaystyle\\ldots,\\cos\(\\omega\_\{x,r\}^\{\\top\}u\+ωt,r⊤v\),sin\(ωx,r⊤u\+ωt,r⊤v\)\]∈ℝ2​r\.\\displaystyle\\omega\_\{t,r\}^\{\\top\}v\),\\sin\(\\omega\_\{x,r\}^\{\\top\}u\+\\omega\_\{t,r\}^\{\\top\}v\)\\Bigr\]\\in\\mathbb\{R\}^\{2r\}\.\(22\)LetΦ∈ℝn×2​r\\Phi\\in\\mathbb\{R\}^\{n\\times 2r\}contain rowsΦi:=φ​\(ui,vi\)⊤\\Phi\_\{i:\}=\\varphi\(u\_\{i\},v\_\{i\}\)^\{\\top\}\. The approximate kernel matrix is

K~\\displaystyle\\widetilde\{K\}=Φ​Φ⊤whereK~i​j=⟨φ​\(ui,vi\),φ​\(uj,vj\)⟩\.\\displaystyle=\\Phi\\Phi^\{\\top\}\\quad\\text\{where\}\\quad\\widetilde\{K\}\_\{ij\}=\\langle\\varphi\(u\_\{i\},v\_\{i\}\),\\varphi\(u\_\{j\},v\_\{j\}\)\\rangle\.\(23\)Next, we show a theoretical guarantee supporting scalable multi\-modal KODA via the joint random Fourier feature implementation:

###### Theorem 5\.1\.

LetKKbe defined in \([20](https://arxiv.org/html/2606.04180#S5.E20)\) andK~\\widetilde\{K\}in \([23](https://arxiv.org/html/2606.04180#S5.E23)\)\. Assume\|Ki​j\|≤1\|K\_\{ij\}\|\\leq 1for alli,ji,j\. Then for anyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta,

‖1n​K~−1n​K‖F≤2\+8​log⁡\(1/δ\)r\\displaystyle\\bigl\\\|\\frac\{1\}\{n\}\\widetilde\{K\}\-\\frac\{1\}\{n\}K\\bigr\\\|\_\{F\}\\;\\leq\\;\\frac\{2\+\\sqrt\{8\\log\(1/\\delta\)\}\}\{\\sqrt\{r\}\}\(24\)Moreover, for anyqqwith eigengapΔq​\(K\)=λq​\(K\)−λq\+1​\(K\)\>0\\Delta\_\{q\}\(K\)=\\lambda\_\{q\}\(K\)\-\\lambda\_\{q\+1\}\(K\)\>0, lettingUUandU~\\widetilde\{U\}denote the top\-qqeigenspaces ofKKandK~\\widetilde\{K\}respectively,

‖sin⁡Θ​\(U~,U\)‖F≤‖K~−K‖FΔq​\(K\)\.\\displaystyle\\bigl\\\|\\sin\\Theta\(\\widetilde\{U\},U\)\\bigr\\\|\_\{F\}\\;\\leq\\;\\frac\{\\\|\\widetilde\{K\}\-K\\\|\_\{F\}\}\{\\Delta\_\{q\}\(K\)\}\.\(25\)

###### Proof\.

We present the proof in the Appendix\. ∎

Theorem[5\.1](https://arxiv.org/html/2606.04180#S5.Thmtheorem1)provides justification for usingK~=Φ​Φ⊤\\widetilde\{K\}=\\Phi\\Phi^\{\\top\}as a proxy for the true product kernel matrixKKin multi\-modal KODA\. In particular,K~\\widetilde\{K\}concentrates aroundKKat rater−1/2r^\{\-1/2\}in Frobenius norm, and the leading eigenspaces are stable wheneverΔq​\(K\)\\Delta\_\{q\}\(K\)is non\-negligible\. This enables covariance/feature\-space implementations whose complexity depends on the joint random\-feature dimension2​r2r, avoiding explicit tensor\-product feature maps of sizedx​dtd\_\{x\}d\_\{t\}\.

## 6Numerical Results

![Refer to caption](https://arxiv.org/html/2606.04180v1/x2.png)

Figure 2:Left: Visualization of the top\-6 discrepancy directions that are strongly grouped under DINOv2 while being weakly clustered under CLIP on the FFHQ dataset, discovered by KODA\. Right: t\-SNE visualization of Top\-10 directions together with clustering scores\.In this section, we evaluate KODA through two complementary tasks\. The first task,*contrastive embedding clustering*, asks whether KODA can identify sample groups that are coherently clustered under one embedding but weakly clustered under another\. The second task,*contrastive embedding alignment*, asks whether the discovered contrastive clusters can be used as actionable slices for targeted alignment between embeddings\. Finally, we provide ablation studies on key design choices in KODA\.

Datasets\.We evaluate our method on a diverse collection of image\-only and image–text datasets to assess discrepancy discovery under both unimodal and multimodal settings\. For image\-only experiments, we use AFHQ\(Choiet al\.,[2020](https://arxiv.org/html/2606.04180#bib.bib14)\), FFHQ\(Karraset al\.,[2019](https://arxiv.org/html/2606.04180#bib.bib15)\), and ImageNet\(Denget al\.,[2009](https://arxiv.org/html/2606.04180#bib.bib16)\)\. For multimodal experiments, we adopt standard image\-caption datasets MSCOCO\(Linet al\.,[2014](https://arxiv.org/html/2606.04180#bib.bib12)\)\.

Models\.For unimodal discrepancy analysis, we consider two widely adopted visual encoders, DINOv2\(Oquabet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib7)\)and CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib1)\)\. For multimodal experiments, we evaluate a diverse set of vision\-language models, including BLIP\(Liet al\.,[2022](https://arxiv.org/html/2606.04180#bib.bib3)\), CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib1)\), OpenCLIP\(Ilharcoet al\.,[2021](https://arxiv.org/html/2606.04180#bib.bib6)\), SigLIP\(Zhaiet al\.,[2023](https://arxiv.org/html/2606.04180#bib.bib8)\), and SigLIP2\(Tschannenet al\.,[2025](https://arxiv.org/html/2606.04180#bib.bib33)\)\.

Implementation details\.All experiments are conducted using the covariance\-operator formulation of KODA, with spectral computations solved via Cholesky decomposition\. We adopt Gaussian \(RBF\) kernels and kernel bandwidths are selected following prior work\(Zhanget al\.,[2024](https://arxiv.org/html/2606.04180#bib.bib42)\)\(Jalaliet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib9)\)to ensure comparable scaling across embeddings\. Further implementation details are provided in[C\.1](https://arxiv.org/html/2606.04180#A3.SS1)\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x3.png)

Figure 3:Multimodal discrepancy analysis on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-1 discrepancy direction identified by KODA for different vision–language models relative to CLIP\.Bottom:Generalized Rayleigh quotient of the identified discrepancy directions under varying constraint quantiles defined on the CLIP kernel\.Contrastive embedding clustering in unimodal encoders\.We begin with the contrastive embedding clustering task for two image encoders, DINOv2 and CLIP, on the AFHQ and FFHQ datasets\.[Figure2](https://arxiv.org/html/2606.04180#S6.F2)illustrates the dominant discrepancy directions identified by KODA that are strongly grouped under DINOv2 while being weakly clustered under CLIP\. For visualization, we select 50 representative samples per direction and project their embeddings using t\-SNE\(Van der Maaten and Hinton,[2008](https://arxiv.org/html/2606.04180#bib.bib27)\)under each model\. To quantify the identified directions, we runkk\-means on the corresponding embeddings with 10 times and report the averaged Adjusted Mutual Information\(Vinhet al\.,[2009](https://arxiv.org/html/2606.04180#bib.bib26)\)\(AMI\) and Normalized Mutual Information\(McDaidet al\.,[2013](https://arxiv.org/html/2606.04180#bib.bib25)\)\(NMI\) between thekk\-means labels and the KODA\-discovered labels\. The results on AFHQ datasets are provided in[Figure7](https://arxiv.org/html/2606.04180#A3.F7)and[Figure8](https://arxiv.org/html/2606.04180#A3.F8)\.

Consistency with reference discrepancy structures\.We further examine whether the discrepancy directions identified by KODA align with semantic mismatches derived from representation similarity statistics\. We use the ImageNet dog breeds dataset since it provides category labels that enable explicit verification\. We derive ground\-truth discrepancy labels based on aggregated similarity statistics\. Without using any label information, KODA recovers dominant discrepancy directions that closely correspond to these mismatched categories as shown in[Figure6](https://arxiv.org/html/2606.04180#A3.F6)\. Additional results and visualizations are provided in Appendix[C\.2](https://arxiv.org/html/2606.04180#A3.SS2)\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x4.png)

Figure 4:t\-SNE visualization of KODA\-selected samples before and after contrastive embedding alignment\.Table 1:Cross\-modal retrieval on the full MSCOCO set and the KODA\-selected subset\. The KODA\-selected subset amplifies the performance gap between CLIP and SigLIP\.Contrastive embedding clustering in vision–language models\.We next evaluate KODA on contrastive embedding clustering for paired image–text data\. All multimodal experiments are conducted on MSCOCO using the joint image–text representation described in[Section5\.2](https://arxiv.org/html/2606.04180#S5.SS2)\. We consider a set of widely used vision–language models, including BLIP, CLIP, OpenCLIP, SigLIP, and SigLIP2, and perform pairwise discrepancy analysis\. As shown in Figure[3](https://arxiv.org/html/2606.04180#S6.F3)\(top\), the resulting samples exhibit distinct multimodal patterns across different models by fixing CLIP as the reference model\. Additional comparing results and visualization across different models are provided in Appendix[C\.4](https://arxiv.org/html/2606.04180#A3.SS4)\.

Quantifying Directional Asymmetry via the Generalized Rayleigh Quotient\.We further quantify the strength of multimodal discrepancy directions using the generalized Rayleigh quotientx⊤​K1​xx⊤​K2​x\\frac\{x^\{\\top\}K\_\{1\}x\}\{x^\{\\top\}K\_\{2\}x\}, whereK1K\_\{1\}andK2K\_\{2\}are normalized RBF kernel matrices induced by the two embeddings\. Sincex⊤​K​xx^\{\\top\}Kxmeasures how strongly directionxxis expressed under kernelKK, larger quotient values indicate stronger directional asymmetry, i\.e\., directions emphasized byK1K\_\{1\}but suppressed byK2K\_\{2\}\. Following Eq\. \(7\), the constraint parameterϵ\\epsilonis set*implicitly*via a quantileq∈\{0\.1,0\.2,…,1\.0\}q\\in\\\{0\.1,0\.2,\\ldots,1\.0\\\}of the eigenvalue distribution ofK2K\_\{2\}, whereϵ\\epsiloncorresponds to theqq\-quantile ofK2K\_\{2\}’s eigenvalues\. Figure[3](https://arxiv.org/html/2606.04180#S6.F3)\(bottom\) reports the quotient values for the Top\-1 discrepancy direction as well as the averages over the Top\-3 and Top\-5 directions\.

Contrastive embedding alignment using KODA\-identified samples\.To examine whether the discovered discrepancy slices are useful beyond visualization, we use them for targeted embedding alignment\. In the unimodal setting, KODA identifies slices that are weakly grouped by CLIP but strongly grouped by DINOv2 on FFHQ dataset\. We fine\-tune CLIP on these selected samples to align its local geometry with DINOv2, following a kernel\-based embedding\-alignment objective of\(Gonget al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib35)\)\. Table[2](https://arxiv.org/html/2606.04180#S6.T2)shows that the aligned CLIP substantially improves its agreement with the KODA\-discovered grouping and approaches the DINOv2 geometry on these slices\. We observe a similar trend in the multimodal setting\. On MSCOCO, KODA identifies image–caption pairs for which BLIP forms a clearer joint structure than CLIP\. The corresponding t\-SNE visualization in[Figure4](https://arxiv.org/html/2606.04180#S6.F4)further shows that the aligned embedding forms a geometry closer to the target embedding on the same selected slice\.

Table 2:Contrastive embedding alignment on KODA\-selected slices\. AMI/NMI/ARI measure agreement between the embedding\-induced clusters and the KODA\-discovered grouping\.Multimodal discrepancy reflects cross\-modal alignment differences\.We further examine whether the multimodal discrepancies found by KODA mainly arise from cross\-modal alignment differences or from general inter\-model mismatch\. Comparing image\-only and joint image–text KODA directions, we find that image\-only embeddings produce noisier directions with weaker semantic separation, while joint image\-text embeddings yield more coherent and concentrated clusters\. Visual comparisons are in[Figures10](https://arxiv.org/html/2606.04180#A3.F10)and[11](https://arxiv.org/html/2606.04180#A3.F11)\. We also evaluate image\-to\-text and text\-to\-image retrieval on MSCOCO, comparing the KODA\-selected subset with the full dataset\. As in[Table1](https://arxiv.org/html/2606.04180#S6.T1), the KODA\-selected slice amplifies the retrieval gap between SigLIP and CLIP\.

Ablation Study\.To examine the sensitivity of KODA to major design parameters, we conduct ablation studies on the number of random Fourier features, the reference sample size, and the kernel function, discussed in Appendix[C\.5](https://arxiv.org/html/2606.04180#A3.SS5)\.

## 7Conclusion and Limitations

In this work, we introduced*Contrastive Embedding Clustering*as a task for identifying sample\-level structures that are organized differently across two embedding representations, and proposed KODA as a constrained kernel\-based framework for solving this task\. By formulating embedding comparison as a quadratic optimization problem with an explicit constraint on weak clusterability under a reference embedding, KODA directly localizes discrepancy directions that are strongly clustered in one embedding but diffuse or suppressed in the other, going beyond global kernel\-difference spectral comparisons\. We showed that the resulting non\-convex problem admits efficient eigenvector\-based characterization and developed scalable implementations through covariance\-block reductions and random feature approximations\. Across experiments, KODA identified fine\-grained and interpretable contrastive clusters across multi\-modal and uni\-modal embeddings\. As limitations, KODA requires a shared reference dataset and focuses on unsupervised discrepancy discovery through kernel\-induced geometry; extending the framework to unmatched datasets, supervised or task\-conditioned discrepancy notions, and stronger statistical guarantees for cluster interpretation are relevant directions for future work\.

## Acknowledgments

This work is supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14210725, and is also supported by CUHK Direct Research Grant with CUHK Project No\. 4055164\. The work is partially supported by a grant under 1\+1\+1 CUHK\-CUHK\(SZ\)\-GDSTC Joint Collaboration Fund\. Also, the authors acknowledge the support from the Hong Kong Research Grants Council \(RGC\) and the Hong Kong PhD Fellowship Scheme \(HKPFS\) award supporting Youqi Wu’s research\. Finally, the authors sincerely thank the anonymous reviewers and meta\-reviewer for their insightful suggestions and constructive feedback\.

## Impact Statement

This work develops methods for comparing multi\-modal embedding representations by identifying dataset\-level discrepancy patterns between models\. A positive impact is to support transparency and interpretability in the evaluation of widely used embedding interfaces, enabling more informed model selection, debugging, and analysis beyond aggregate benchmark metrics\. As with other representation analysis tools, such methods could also be misused to exploit model\-specific weaknesses or to support undesirable downstream applications if applied without appropriate safeguards\. We therefore emphasize that KODA is intended for controlled evaluation and auditing purposes on shared reference datasets, and we encourage careful consideration of dataset provenance, privacy, and downstream use when applying embedding comparison techniques in practice\.

## References

- D\. Bau, B\. Zhou, A\. Khosla, A\. Oliva, and A\. Torralba \(2017\)Network dissection: quantifying interpretability of deep visual representations\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 3319–3327\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2017.354),[Link](https://doi.org/10.1109/CVPR.2017.354)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p2.1)\.
- M\. Bińkowski, D\. J\. Sutherland, M\. Arbel, and A\. Gretton \(2018\)Demystifying MMD GANs\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- M\. Cherti, R\. Beaumont, R\. Wightman, M\. Wortsman, G\. Ilharco, C\. Gordon, C\. Schuhmann, L\. Schmidt, and J\. Jitsev \(2023\)Reproducible scaling laws for contrastive language\-image learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2818–2829\.Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p1.1),[§2](https://arxiv.org/html/2606.04180#S2.p1.1)\.
- R\. Chitta, R\. Jin, and A\. K\. Jain \(2012\)Efficient kernel clustering using random fourier features\.In2012 IEEE 12th International Conference on Data Mining,pp\. 161–170\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- Y\. Choi, Y\. Uh, J\. Yoo, and J\. Ha \(2020\)Stargan v2: diverse image synthesis for multiple domains\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 8188–8197\.Cited by:[§C\.3](https://arxiv.org/html/2606.04180#A3.SS3.p1.11),[§6](https://arxiv.org/html/2606.04180#S6.p2.1)\.
- M\. Darrin, P\. Formont, I\. Ayed, J\. C\. Cheung, and P\. Piantanida \(2024\)When is an embedding model more promising than another?\.Advances in Neural Information Processing Systems37,pp\. 68330–68379\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p1.1)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)Imagenet: a large\-scale hierarchical image database\.In2009 IEEE conference on computer vision and pattern recognition,pp\. 248–255\.Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p2.1)\.
- D\. Friedman and A\. B\. Dieng \(2023\)The Vendi score: a diversity evaluation metric for machine learning\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- D\. Gedon, A\. H\. Ribeiro, N\. Wahlström, and T\. B\. Schön \(2023\)Invertible kernel PCA with random fourier features\.IEEE Signal Processing Letters30,pp\. 563–567\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- M\. Ghashami, D\. J\. Perry, and J\. Phillips \(2016\)Streaming kernel principal component analysis\.InArtificial intelligence and statistics,pp\. 1365–1374\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- A\. Ghorbani, J\. Wexler, J\. Y\. Zou, and B\. Kim \(2019\)Towards automatic concept\-based explanations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32,pp\. 9277–9286\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p2.1)\.
- S\. Gong, Y\. Jiang, Q\. Dou, and F\. Farnia \(2025a\)Kernel\-based unsupervised embedding alignment for enhanced visual representation in vision\-language models\.InInternational Conference on Machine Learning,pp\. 19912–19931\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1),[§6](https://arxiv.org/html/2606.04180#S6.p9.1)\.
- S\. Gong, H\. Lei, Q\. Dou, and F\. Farnia \(2025b\)Boosting the visual interpretability of CLIP via adversarial fine\-tuning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=khuIvzxPRp)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p2.1)\.
- A\. Gretton, K\. M\. Borgwardt, M\. J\. Rasch, B\. Schölkopf, and A\. Smola \(2012\)A kernel two\-sample test\.Journal of Machine Learning Research13\(25\),pp\. 723–773\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- X\. Hu, H\. Leung, and F\. Farnia \(2025a\)A multi\-armed bandit approach to online selection and evaluation of generative models\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 1864–1872\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- X\. Hu, H\. Leung, and F\. Farnia \(2025b\)PAK\-UCB contextual bandit: an online learning approach to prompt\-aware selection of generative models and LLMs\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 24447–24481\.External Links:[Link](https://proceedings.mlr.press/v267/hu25m.html)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- M\. Huh, B\. Cheung, T\. Wang, and P\. Isola \(2024\)Position: the platonic representation hypothesis\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 20617–20642\.External Links:[Link](https://proceedings.mlr.press/v235/huh24a.html)Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p1.1)\.
- G\. Ilharco, M\. Wortsman, R\. Wightman, C\. Gordon, N\. Carlini, R\. Taori, A\. Dave, V\. Shankar, H\. Namkoong, J\. Miller, H\. Hajishirzi, A\. Farhadi, and L\. Schmidt \(2021\)OpenCLIPNote:If you use this software, please cite it as below\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.5143773),[Link](https://doi.org/10.5281/zenodo.5143773)Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- D\. Jafari and F\. Farnia \(2026\)DAK\-UCB: diversity\-aware prompt routing for LLMs and generative models\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=nnN2TKlS5C)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- M\. Jalali, B\. Dibaei Nia, and F\. Farnia \(2025a\)Towards an explainable comparison and alignment of feature embeddings\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 26757–26796\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1),[§C\.1](https://arxiv.org/html/2606.04180#A3.SS1.p1.5),[§C\.3](https://arxiv.org/html/2606.04180#A3.SS3.p1.11),[§1](https://arxiv.org/html/2606.04180#S1.p3.1),[§2](https://arxiv.org/html/2606.04180#S2.p3.1),[§6](https://arxiv.org/html/2606.04180#S6.p4.1)\.
- M\. Jalali, H\. Lei, A\. Gohari, and F\. Farnia \(2025b\)SPARKE: scalable prompt\-aware diversity and novelty guidance in diffusion models via RKE score\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- M\. Jalali, C\. T\. Li, and F\. Farnia \(2023\)An information\-theoretic evaluation of generative models in learning multi\-modal distributions\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- M\. Jalali, A\. Ospanov, A\. Gohari, and F\. Farnia \(2026\)Conditional vendi score: prompt\-aware diversity evaluation for generative ai models and llms\.InProceedings of The 29th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- C\. Jia, Y\. Yang, Y\. Xia, Y\. Chen, Z\. Parekh, H\. Pham, Q\. Le, Y\. Sung, Z\. Li, and T\. Duerig \(2021\)Scaling up visual and vision\-language representation learning with noisy text supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 4904–4916\.External Links:[Link](https://proceedings.mlr.press/v139/jia21b.html)Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p1.1),[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§2](https://arxiv.org/html/2606.04180#S2.p1.1)\.
- T\. Karras, S\. Laine, and T\. Aila \(2019\)A style\-based generator architecture for generative adversarial networks\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 4401–4410\.Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p2.1)\.
- B\. Kim, M\. Wattenberg, J\. Gilmer, C\. Cai, J\. Wexler, F\. Viegas, and R\. Sayres \(2018\)Interpretability beyond feature attribution: quantitative testing with concept activation vectors \(TCAV\)\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 2668–2677\.External Links:[Link](https://proceedings.mlr.press/v80/kim18d.html)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p2.1)\.
- P\. W\. Koh, T\. Nguyen, Y\. S\. Tang, S\. Mussmann, E\. Pierson, B\. Kim, and P\. Liang \(2020\)Concept bottleneck models\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.119,pp\. 5338–5348\.External Links:[Link](https://proceedings.mlr.press/v119/koh20a.html)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p2.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023\)BLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 19730–19742\.Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p1.1),[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§2](https://arxiv.org/html/2606.04180#S2.p1.1)\.
- J\. Li, D\. Li, C\. Xiong, and S\. Hoi \(2022\)BLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 12888–12900\.Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p1.1),[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§2](https://arxiv.org/html/2606.04180#S2.p1.1),[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- T\. Lin, M\. Maire, S\. J\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. Zitfnick \(2014\)Microsoft coco: common objects in context\.InComputer Vision – ECCV 2014,pp\. 740–755\.Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§6](https://arxiv.org/html/2606.04180#S6.p2.1)\.
- A\. F\. McDaid, D\. Greene, and N\. Hurley \(2013\)Normalized mutual information to evaluate overlapping community finding algorithms\.External Links:1110\.2515Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p5.2)\.
- A\. Y\. Ng, M\. I\. Jordan, and Y\. Weiss \(2001\)On spectral clustering: analysis and an algorithm\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby,et al\.\(2023\)Dinov2: learning robust visual features without supervision\.arXiv preprint arXiv:2304\.07193\.Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- A\. Ospanov, M\. Jalali, and F\. Farnia \(2025\)Scendi score: prompt\-aware diversity evaluation via schur complement of clip embeddings\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 16927–16937\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- A\. Ospanov, J\. Zhang, M\. Jalali, X\. Cao, A\. Bogdanov, and F\. Farnia \(2024\)Towards a scalable reference\-free evaluation of generative models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2606.04180#S1.p1.1),[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§2](https://arxiv.org/html/2606.04180#S2.p1.1),[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- A\. Rahimi and B\. Recht \(2007\)Random features for large\-scale kernel machines\.InAdvances in Neural Information Processing Systems,pp\. 1177–1184\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1),[§1](https://arxiv.org/html/2606.04180#S1.p6.1),[§3\.3](https://arxiv.org/html/2606.04180#S3.SS3.p1.5)\.
- P\. Rezaei, F\. Farnia, and C\. T\. Li \(2025\)Be more diverse than the most diverse: optimal mixtures of generative models\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=2Chkk5Ye2s)Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- B\. Schölkopf, A\. J\. Smola, and K\. Müller \(1998\)Nonlinear component analysis as a kernel eigenvalue problem\.Neural Computation10\(5\),pp\. 1299–1319\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- B\. K\. Sriperumbudur and N\. Sterge \(2022\)Approximate kernel PCA: computational versus statistical trade\-off\.The Annals of Statistics50\(5\),pp\. 2713–2736\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- G\. Stein, J\. C\. Cresswell, R\. Hosseinzadeh, Y\. Sui, B\. L\. Ross, V\. Villecroze, Z\. Liu, A\. L\. Caterini, J\. E\. T\. Taylor, and G\. Loaiza\-Ganem \(2023\)Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 3732–3784\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- D\. J\. Sutherland and J\. Schneider \(2015\)On the error of random fourier features\.arXiv preprint arXiv:1506\.02785\.Cited by:[§3\.3](https://arxiv.org/html/2606.04180#S3.SS3.p1.5)\.
- D\. J\. Sutherland, H\. Strathmann, M\. Arbel, and A\. Gretton \(2018\)Efficient and principled score estimation with nyström kernel exponential families\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 652–660\.Cited by:[§B\.4](https://arxiv.org/html/2606.04180#A2.SS4.p4.5),[§B\.5](https://arxiv.org/html/2606.04180#A2.SS5.p3.5)\.
- M\. Tschannen, A\. Gritsenko, X\. Wang, M\. F\. Naeem, I\. Alabdulmohsin, N\. Parthasarathy, T\. Evans, L\. Beyer, Y\. Xia, B\. Mustafa,et al\.\(2025\)Siglip 2: multilingual vision\-language encoders with improved semantic understanding, localization, and dense features\.arXiv preprint arXiv:2502\.14786\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p1.1),[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- E\. Ullah, P\. Mianjy, T\. V\. Marinov, and R\. Arora \(2018\)Streaming kernel PCA withO​\(n\)O\(\\sqrt\{n\}\)random features\.Advances in Neural Information Processing Systems31\.Cited by:[Appendix A](https://arxiv.org/html/2606.04180#A1.p2.1)\.
- L\. Van der Maaten and G\. Hinton \(2008\)Visualizing data using t\-sne\.\.Journal of machine learning research9\(11\)\.Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p5.2)\.
- N\. X\. Vinh, J\. Epps, and J\. Bailey \(2009\)Information theoretic measures for clusterings comparison: is a correction for chance necessary?\.InProceedings of the 26th annual international conference on machine learning,pp\. 1073–1080\.Cited by:[§6](https://arxiv.org/html/2606.04180#S6.p5.2)\.
- Z\. Wang, F\. Farzan, Z\. Lin, Y\. Shen, and B\. Yu \(2025\)On the distributed evaluation of generative models\.InProceedings of the IEEE/CVF International Conference on Computer Vision Workshops,pp\. 7644–7653\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- Y\. Wu and F\. Farnia \(2026\)The maximum von neumann entropy principle: theory and applications in machine learning\.InIEEE International Symposium on Information Theory \(ISIT\),Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- Y\. Wu, J\. Zhang, and F\. Farnia \(2025\)When kernels multiply, clusters unify: fusing embeddings with the kronecker product\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p1.1),[§6](https://arxiv.org/html/2606.04180#S6.p3.1)\.
- J\. Zhang, M\. Jalali, C\. T\. Li, and F\. Farnia \(2025\)Unveiling differences in generative models: a scalable differential clustering approach\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8269–8278\.Cited by:[§2](https://arxiv.org/html/2606.04180#S2.p3.1)\.
- J\. Zhang, C\. T\. Li, and F\. Farnia \(2024\)An interpretable evaluation of entropy\-based novelty of generative models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 59148–59172\.Cited by:[§C\.1](https://arxiv.org/html/2606.04180#A3.SS1.p1.5),[§2](https://arxiv.org/html/2606.04180#S2.p3.1),[§6](https://arxiv.org/html/2606.04180#S6.p4.1)\.

## Appendix AAdditional Related Works

Beyond downstream benchmarks: representation\-level comparison\.Recent work has explored task\-agnostic viewpoints on embedding comparison that deviate from pure downstream evaluation\.Darrinet al\.\([2024](https://arxiv.org/html/2606.04180#bib.bib11)\)propose an information\-theoretic approach to comparing embedding models based on notions of sufficiency/informativeness, enabling comparison without labeled tasks\. The Platonic Representation Hypothesis\(Huhet al\.,[2024](https://arxiv.org/html/2606.04180#bib.bib10)\)studies the extent to which independently trained models share representational structure, offering a complementary lens on when embeddings may share geometric organization\.

Kernel and spectral tools\.Kernel spectral methods provide classical machinery for studying grouping structure through eigensystemsChittaet al\.\([2012](https://arxiv.org/html/2606.04180#bib.bib18)\); Ghashamiet al\.\([2016](https://arxiv.org/html/2606.04180#bib.bib19)\); Ullahet al\.\([2018](https://arxiv.org/html/2606.04180#bib.bib20)\); Sriperumbudur and Sterge \([2022](https://arxiv.org/html/2606.04180#bib.bib21)\); Gedonet al\.\([2023](https://arxiv.org/html/2606.04180#bib.bib22)\)\. Kernel PCA\(Schölkopfet al\.,[1998](https://arxiv.org/html/2606.04180#bib.bib23)\)and spectral clustering\(Nget al\.,[2001](https://arxiv.org/html/2606.04180#bib.bib24)\)relate eigenstructure of similarity matrices to latent clusters, while Random Fourier Features\(Rahimi and Recht,[2007](https://arxiv.org/html/2606.04180#bib.bib17)\)enable scalable approximations for shift\-invariant kernels\. Building on these tools, spectral kernel\-based embedding comparison methods have been proposed, including analyzing kernel differences constructed from a shared reference dataset\(Jalaliet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib9)\)\. Our work follows this kernel\-based, reference\-set comparison paradigm but adopts an optimization\-based formulation that explicitly enforces weak grouping under one embedding while maximizing grouping strength under another\.

## Appendix BProofs

### B\.1Proof of Proposition[4\.1](https://arxiv.org/html/2606.04180#S4.Thmtheorem1)

Consider Lagrangian multipliersλ≥0\\lambda\\geq 0forx⊤​KB​x≤ϵx^\{\\top\}K\_\{B\}\\,x\\leq\\epsilonandν∈ℝ\\nu\\in\\mathbb\{R\}for‖x‖22=1\\\|x\\\|\_\{2\}^\{2\}=1, and consider

ℒ​\(x,λ,ν\)=x⊤​KA​x−λ​\(x⊤​KB​x−ϵ\)−ν​\(x⊤​x−1\)\.\\mathcal\{L\}\(x,\\lambda,\\nu\)=x^\{\\top\}K\_\{A\}\\,x\-\\lambda\(x^\{\\top\}K\_\{B\}\\,x\-\\epsilon\)\-\\nu\(x^\{\\top\}x\-1\)\.At a KKT point\(x⋆,λ⋆,ν⋆\)\(x^\{\\star\},\\lambda^\{\\star\},\\nu^\{\\star\}\), stationarity gives2​\(KA−λ⋆​KB−ν⋆​I\)​x⋆=02\(K\_\{A\}\\,\-\\lambda^\{\\star\}K\_\{B\}\\,\-\\nu^\{\\star\}I\)x^\{\\star\}=0, yielding \([9](https://arxiv.org/html/2606.04180#S4.E9)\); complementary slackness yields \([10](https://arxiv.org/html/2606.04180#S4.E10)\)\. The strict feasibility assumption is a standard constraint qualification for the inequality constraint on the sphere\.

### B\.2Proof of Proposition[4\.2](https://arxiv.org/html/2606.04180#S4.Thmtheorem2)

First, note thatUt−1⊤​x=0U\_\{t\-1\}^\{\\top\}x=0is equivalent tox=Pt−1​xx=P\_\{t\-1\}x, which directly follows from the definition ofPt−1P\_\{t\-1\}\. For suchxxand any symmetricDD, we havex⊤​D​x=x⊤​\(Pt−1​D​Pt−1\)​xx^\{\\top\}Dx=x^\{\\top\}\(P\_\{t\-1\}DP\_\{t\-1\}\)xsincePt−1=Pt−1⊤=Pt−12P\_\{t\-1\}=P\_\{t\-1\}^\{\\top\}=P\_\{t\-1\}^\{2\}\. Applying this toD=KA,KBD=K\_\{A\}\\,,K\_\{B\}\\,yields the equivalence\.

### B\.3Proof of Proposition[4\.3](https://arxiv.org/html/2606.04180#S4.Thmtheorem3)

First note that

KA−λ​KB\\displaystyle K\_\{A\}\\,\-\\lambda K\_\{B\}\\,=ΦA​ΦA⊤−λ​ΦB​ΦB⊤=Φ​Sλ​Φ⊤\.\\displaystyle=\\Phi\_\{A\}\\Phi\_\{A\}^\{\\top\}\-\\lambda\\Phi\_\{B\}\\Phi\_\{B\}^\{\\top\}=\\Phi S\_\{\\lambda\}\\Phi^\{\\top\}\.For anyu∈ℝd1\+d2u\\in\\mathbb\{R\}^\{d\_\{1\}\+d\_\{2\}\},

\(KA−λ​KB\)​\(Φ​u\)\\displaystyle\(K\_\{A\}\\,\-\\lambda K\_\{B\}\\,\)\(\\Phi u\)=Φ​Sλ​Φ⊤​Φ​u=Φ​Sλ​\(Φ⊤​Φ\)​u=Φ​Mλ​u\.\\displaystyle=\\Phi S\_\{\\lambda\}\\Phi^\{\\top\}\\Phi u=\\Phi S\_\{\\lambda\}\(\\Phi^\{\\top\}\\Phi\)u=\\Phi M\_\{\\lambda\}u\.Hence, ifMλ​uλ=ηλ​uλM\_\{\\lambda\}u\_\{\\lambda\}=\\eta\_\{\\lambda\}u\_\{\\lambda\}, then withxλ=Φ​uλx\_\{\\lambda\}=\\Phi u\_\{\\lambda\}we have

\(KA−λ​KB\)​xλ=ηλ​xλ,\(K\_\{A\}\\,\-\\lambda K\_\{B\}\\,\)x\_\{\\lambda\}=\\eta\_\{\\lambda\}x\_\{\\lambda\},thusxλx\_\{\\lambda\}is an eigenvector ofKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,with eigenvalueηλ\\eta\_\{\\lambda\}\. Moreover,ηλ≠0\\eta\_\{\\lambda\}\\neq 0impliesxλ≠0x\_\{\\lambda\}\\neq 0: ifΦ​uλ=0\\Phi u\_\{\\lambda\}=0thenG​uλ=Φ⊤​Φ​uλ=0Gu\_\{\\lambda\}=\\Phi^\{\\top\}\\Phi u\_\{\\lambda\}=0and thusMλ​uλ=Sλ​G​uλ=0M\_\{\\lambda\}u\_\{\\lambda\}=S\_\{\\lambda\}Gu\_\{\\lambda\}=0, forcingηλ=0\\eta\_\{\\lambda\}=0\.

It remains to show thatηλ\\eta\_\{\\lambda\}is also an eigenvalue ofMλM\_\{\\lambda\}\. This follows from the fact that for every matrix pairA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}andB∈ℝn×mB\\in\\mathbb\{R\}^\{n\\times m\}\(for any integersm,nm,n\),A​BABandB​ABAshare the same non\-zero eigenvalues \(including multiplicities\)\. Applying this to the caseA=ΦA=\\PhiandB=Sλ​Φ⊤B=S\_\{\\lambda\}\\Phi^\{\\top\}shows thatΦ​Sλ​Φ⊤=KA−λ​KB\\Phi S\_\{\\lambda\}\\Phi^\{\\top\}=K\_\{A\}\\,\-\\lambda K\_\{B\}\\,andSλ​Φ⊤​Φ=MλS\_\{\\lambda\}\\Phi^\{\\top\}\\Phi=M\_\{\\lambda\}share the same non\-zero eigenvalues\. SinceKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,is symmetric, its eigenvalues are real, and particularly its largest eigenvalueηλ=λmax​\(KA−λ​KB\)\\eta\_\{\\lambda\}=\\lambda\_\{\\max\}\(K\_\{A\}\\,\-\\lambda K\_\{B\}\\,\)is among the eigenvalues ofMλM\_\{\\lambda\}\. Therefore,uλu\_\{\\lambda\}can be chosen as an eigenvector ofMλM\_\{\\lambda\}associated withηλ\\eta\_\{\\lambda\}, and the lifted vectorxλ=Φ​uλx\_\{\\lambda\}=\\Phi u\_\{\\lambda\}is a principal eigenvector ofKA−λ​KBK\_\{A\}\\,\-\\lambda K\_\{B\}\\,\.

### B\.4Proof of Theorem[4\.4](https://arxiv.org/html/2606.04180#S4.Thmtheorem4)

LetD:=d1\+d2D:=d\_\{1\}\+d\_\{2\}\. Definez​\(X\):=\[ϕ1​\(X\);ϕ2​\(X\)\]∈ℝDz\(X\):=\[\\phi\_\{1\}\(X\);\\phi\_\{2\}\(X\)\]\\in\\mathbb\{R\}^\{D\}and assume‖z​\(X\)‖22=‖ϕ1​\(X\)‖22\+‖ϕ2​\(X\)‖22≤2\\big\\\|z\(X\)\\big\\\|\_\{2\}^\{2\}=\\big\\\|\\phi\_\{1\}\(X\)\\big\\\|\_\{2\}^\{2\}\+\\big\\\|\\phi\_\{2\}\(X\)\\big\\\|\_\{2\}^\{2\}\\leq 2almost surely\. Let

G:=𝔼​\[z​\(X\)​z​\(X\)⊤\]⪰0,G^:=1n​∑i=1nz​\(Xi\)​z​\(Xi\)⊤⪰0,G:=\\mathbb\{E\}\[z\(X\)z\(X\)^\{\\top\}\]\\succeq 0,\\qquad\\widehat\{G\}:=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}z\(X\_\{i\}\)z\(X\_\{i\}\)^\{\\top\}\\succeq 0,and forλ≥0\\lambda\\geq 0defineSλ:=diag​\(Id1,−λ​Id2\)S\_\{\\lambda\}:=\\mathrm\{diag\}\(I\_\{d\_\{1\}\},\-\\lambda I\_\{d\_\{2\}\}\),

Bλ:=G1/2​Sλ​G1/2,B^λ:=G^1/2​Sλ​G^1/2\.B\_\{\\lambda\}:=G^\{1/2\}S\_\{\\lambda\}G^\{1/2\},\\qquad\\widehat\{B\}\_\{\\lambda\}:=\\widehat\{G\}^\{1/2\}S\_\{\\lambda\}\\widehat\{G\}^\{1/2\}\.NoteBλB\_\{\\lambda\}andB^λ\\widehat\{B\}\_\{\\lambda\}are symmetric\.

First, we prove a Frobenius concentration result forG^\\widehat\{G\}using the Hilbert\-space Hoeffding’s inequality\. LetWi:=z​\(Xi\)​z​\(Xi\)⊤W\_\{i\}:=z\(X\_\{i\}\)z\(X\_\{i\}\)^\{\\top\}andZi:=Wi−GZ\_\{i\}:=W\_\{i\}\-G, and therefore we have𝔼​\[Zi\]=0\\mathbb\{E\}\[Z\_\{i\}\]=0andG^−G=1n​∑i=1nZi\\widehat\{G\}\-G=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}Z\_\{i\}\. We bound‖Zi‖F\\big\\\|Z\_\{i\}\\big\\\|\_\{F\}almost surely\.

First, for any vectoraa,‖a​a⊤‖F=‖a‖22\\big\\\|aa^\{\\top\}\\big\\\|\_\{F\}=\\big\\\|a\\big\\\|\_\{2\}^\{2\}: indeed,‖a​a⊤‖F2=Tr​\(\(a​a⊤\)⊤​\(a​a⊤\)\)=Tr​\(a​a⊤​a​a⊤\)=‖a‖24\\big\\\|aa^\{\\top\}\\big\\\|\_\{F\}^\{2\}=\\mathrm\{Tr\}\(\(aa^\{\\top\}\)^\{\\top\}\(aa^\{\\top\}\)\)=\\mathrm\{Tr\}\(aa^\{\\top\}aa^\{\\top\}\)=\\big\\\|a\\big\\\|\_\{2\}^\{4\}\. Thus,

‖Wi‖F=‖z​\(Xi\)​z​\(Xi\)⊤‖F=‖z​\(Xi\)‖22≤2a\.s\.\\big\\\|W\_\{i\}\\big\\\|\_\{F\}=\\big\\\|z\(X\_\{i\}\)z\(X\_\{i\}\)^\{\\top\}\\big\\\|\_\{F\}=\\big\\\|z\(X\_\{i\}\)\\big\\\|\_\{2\}^\{2\}\\leq 2\\quad\\text\{a\.s\.\}Also, by Jensen’s inequality and the triangle inequality for the Frobenius norm,

‖G‖F=‖𝔼​\[Wi\]‖F≤𝔼​‖Wi‖F≤2\.\\big\\\|G\\big\\\|\_\{F\}=\\big\\\|\\mathbb\{E\}\[W\_\{i\}\]\\big\\\|\_\{F\}\\leq\\mathbb\{E\}\\big\\\|W\_\{i\}\\big\\\|\_\{F\}\\leq 2\.Therefore,

‖Zi‖F=‖Wi−G‖F≤‖Wi‖F\+‖G‖F≤4a\.s\.\\big\\\|Z\_\{i\}\\big\\\|\_\{F\}=\\big\\\|W\_\{i\}\-G\\big\\\|\_\{F\}\\leq\\big\\\|W\_\{i\}\\big\\\|\_\{F\}\+\\big\\\|G\\big\\\|\_\{F\}\\leq 4\\quad\\text\{a\.s\.\}
We now apply the Hoeffding\-type inequality for random vectors in Hilbert spaces\(Sutherlandet al\.,[2018](https://arxiv.org/html/2606.04180#bib.bib13)\)to the i\.i\.d\. Hilbert\-space\-valued variablesZiZ\_\{i\}in the Hilbert space ofD×DD\\times Dmatrices equipped with Frobenius norm\. WithL=4L=4, we obtain: for anyδ∈\(0,1\)\\delta\\in\(0,1\), with probability at least1−δ1\-\\delta,

∥G^−G∥F=∥1n∑i=1nZi∥F≤4n\(1\+2​log⁡1δ\)=:ηn\(δ\)\.\\displaystyle\\big\\\|\\widehat\{G\}\-G\\big\\\|\_\{F\}=\\Bigl\\\|\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}Z\_\{i\}\\Bigr\\\|\_\{F\}\\leq\\frac\{4\}\{\\sqrt\{n\}\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)=:\\eta\_\{n\}\(\\delta\)\.\(26\)
Subsequently, we apply the Powers–Størmer inequality showing that for PSD matricesA,B⪰0A,B\\succeq 0we have

‖A1/2−B1/2‖F2≤‖A−B‖∗,\\displaystyle\\\|A^\{1/2\}\-B^\{1/2\}\\\|\_\{F\}^\{2\}\\leq\\\|A\-B\\\|\_\{\*\},\(27\)where∥⋅∥∗\\\|\\cdot\\\|\_\{\*\}is the nuclear \(trace\) norm\. Applying \([27](https://arxiv.org/html/2606.04180#A2.E27)\) toA=G^A=\\widehat\{G\}andB=GB=Gyields

‖G^1/2−G1/2‖F≤‖G^−G‖∗1/2\.\\\|\\widehat\{G\}^\{1/2\}\-G^\{1/2\}\\\|\_\{F\}\\leq\\\|\\widehat\{G\}\-G\\\|\_\{\*\}^\{1/2\}\.For anyD×DD\\times DmatrixXX,‖X‖∗≤rank​\(X\)​‖X‖F≤D​‖X‖F\\\|X\\\|\_\{\*\}\\leq\\sqrt\{\\mathrm\{rank\}\(X\)\}\\,\\\|X\\\|\_\{F\}\\leq\\sqrt\{D\}\\,\\\|X\\\|\_\{F\}\. Therefore,

‖G^1/2−G1/2‖F≤D1/4​‖G^−G‖F1/2\.\\displaystyle\\\|\\widehat\{G\}^\{1/2\}\-G^\{1/2\}\\\|\_\{F\}\\leq D^\{1/4\}\\,\\\|\\widehat\{G\}\-G\\\|\_\{F\}^\{1/2\}\.\(28\)
Then, we bound the norm difference‖B^λ−Bλ‖F\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}using‖G‖2≤2\\\|G\\\|\_\{2\}\\leq 2\. To do so, we expand

B^λ−Bλ=\(G^1/2−G1/2\)​Sλ​G^1/2\+G1/2​Sλ​\(G^1/2−G1/2\)\.\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}=\(\\widehat\{G\}^\{1/2\}\-G^\{1/2\}\)S\_\{\\lambda\}\\widehat\{G\}^\{1/2\}\+G^\{1/2\}S\_\{\\lambda\}\(\\widehat\{G\}^\{1/2\}\-G^\{1/2\}\)\.Taking the Frobenius norm and using submultiplicativity inequalities‖A​X‖F≤‖A‖2​‖X‖F\\\|AX\\\|\_\{F\}\\leq\\\|A\\\|\_\{2\}\\\|X\\\|\_\{F\}and‖X​B‖F≤‖B‖2​‖X‖F\\\|XB\\\|\_\{F\}\\leq\\\|B\\\|\_\{2\}\\\|X\\\|\_\{F\}result in the following inequality:

‖B^λ−Bλ‖F≤‖Sλ‖2​\(‖G^1/2‖2\+‖G1/2‖2\)​‖G^1/2−G1/2‖F\.\\displaystyle\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}\\leq\\\|S\_\{\\lambda\}\\\|\_\{2\}\\big\(\\\|\\widehat\{G\}^\{1/2\}\\\|\_\{2\}\+\\\|G^\{1/2\}\\\|\_\{2\}\\big\)\\,\\\|\\widehat\{G\}^\{1/2\}\-G^\{1/2\}\\\|\_\{F\}\.\(29\)We now show that‖G^1/2‖2≤2\\\|\\widehat\{G\}^\{1/2\}\\\|\_\{2\}\\leq\\sqrt\{2\}and‖G1/2‖2≤2\\\|G^\{1/2\}\\\|\_\{2\}\\leq\\sqrt\{2\}\.

SinceG⪰0G\\succeq 0,‖G‖2=λmax​\(G\)≤Tr​\(G\)\\\|G\\\|\_\{2\}=\\lambda\_\{\\max\}\(G\)\\leq\\mathrm\{Tr\}\(G\)\. Moreover,

Tr​\(G\)=𝔼​Tr​\(z​\(X\)​z​\(X\)⊤\)=𝔼​‖z​\(X\)‖22≤2\\mathrm\{Tr\}\(G\)=\\mathbb\{E\}\\,\\mathrm\{Tr\}\(z\(X\)z\(X\)^\{\\top\}\)=\\mathbb\{E\}\\\|z\(X\)\\\|\_\{2\}^\{2\}\\leq 2As a result,‖G‖2≤2\\\|G\\\|\_\{2\}\\leq 2and thus‖G1/2‖2=‖G‖2≤2\\\|G^\{1/2\}\\\|\_\{2\}=\\sqrt\{\\\|G\\\|\_\{2\}\}\\leq\\sqrt\{2\}\. Similarly, we haveG^⪰0\\widehat\{G\}\\succeq 0and‖G^‖2≤Tr​\(G^\)\\\|\\widehat\{G\}\\\|\_\{2\}\\leq\\mathrm\{Tr\}\(\\widehat\{G\}\)\. However, note that

Tr​\(G^\)=1n​∑i=1nTr​\(z​\(Xi\)​z​\(Xi\)⊤\)=1n​∑i=1n‖z​\(Xi\)‖22≤2\\mathrm\{Tr\}\(\\widehat\{G\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathrm\{Tr\}\(z\(X\_\{i\}\)z\(X\_\{i\}\)^\{\\top\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\\|z\(X\_\{i\}\)\\\|\_\{2\}^\{2\}\\leq 2which holds deterministically because each‖z​\(Xi\)‖22≤2\\\|z\(X\_\{i\}\)\\\|\_\{2\}^\{2\}\\leq 2\. Therefore,‖G^‖2≤2\\\|\\widehat\{G\}\\\|\_\{2\}\\leq 2and‖G^1/2‖2≤2\\\|\\widehat\{G\}^\{1/2\}\\\|\_\{2\}\\leq\\sqrt\{2\}\. Consequently, the following holds

‖G^1/2‖2\+‖G1/2‖2≤2​2\.\\displaystyle\\\|\\widehat\{G\}^\{1/2\}\\\|\_\{2\}\+\\\|G^\{1/2\}\\\|\_\{2\}\\leq 2\\sqrt\{2\}\.\(30\)
Then, we substitute \([28](https://arxiv.org/html/2606.04180#A2.E28)\) and \([30](https://arxiv.org/html/2606.04180#A2.E30)\) into \([29](https://arxiv.org/html/2606.04180#A2.E29)\):

‖B^λ−Bλ‖F≤2​2​‖Sλ‖2​D1/4​‖G^−G‖F1/2\.\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}\\leq 2\\sqrt\{2\}\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,D^\{1/4\}\\,\\\|\\widehat\{G\}\-G\\\|\_\{F\}^\{1/2\}\.On the event \([26](https://arxiv.org/html/2606.04180#A2.E26)\), we have‖G^−G‖F≤ηn​\(δ\)\\\|\\widehat\{G\}\-G\\\|\_\{F\}\\leq\\eta\_\{n\}\(\\delta\)and hence

‖B^λ−Bλ‖F≤2​2​‖Sλ‖2​D1/4​ηn​\(δ\)1/2\.\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}\\leq 2\\sqrt\{2\}\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,D^\{1/4\}\\,\\eta\_\{n\}\(\\delta\)^\{1/2\}\.Plugging inηn​\(δ\)\\eta\_\{n\}\(\\delta\)from \([26](https://arxiv.org/html/2606.04180#A2.E26)\) leads to

‖B^λ−Bλ‖F≤2​2​‖Sλ‖2​D1/4​\(4n​\(1\+2​log⁡1δ\)\)1/2=8​2​‖Sλ‖2​D1/4​n−1/4​\(1\+2​log⁡1δ\)1/2\.\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}\\leq 2\\sqrt\{2\}\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,D^\{1/4\}\\Bigl\(\\frac\{4\}\{\\sqrt\{n\}\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)\\Bigr\)^\{1/2\}=8\\sqrt\{2\}\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,D^\{1/4\}\\,n^\{\-1/4\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)^\{1/2\}\.Finally, note that the norm inequality∥⋅∥2≤∥⋅∥F\\\|\\cdot\\\|\_\{2\}\\leq\\\|\\cdot\\\|\_\{F\}implies that

‖B^λ−Bλ‖2≤‖B^λ−Bλ‖F≤8​2​‖Sλ‖2​D1/4​n−1/4​\(1\+2​log⁡1δ\)1/2\.\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{2\}\\leq\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{F\}\\leq 8\\sqrt\{2\}\\,\\\|S\_\{\\lambda\}\\\|\_\{2\}\\,D^\{1/4\}\\,n^\{\-1/4\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)^\{1/2\}\.Knowing that8​2<128\\sqrt\{2\}<12, the proof for the matrix concentration is complete\. Finally, becauseBλB\_\{\\lambda\}andB^λ\\widehat\{B\}\_\{\\lambda\}are symmetric matrices, the rank\-one Davis–Kahansin⁡Θ\\sin\\Thetabound applies as follows: Given thatγλ:=λ1​\(Bλ\)−λ2​\(Bλ\)\>0\\gamma\_\{\\lambda\}:=\\lambda\_\{1\}\(B\_\{\\lambda\}\)\-\\lambda\_\{2\}\(B\_\{\\lambda\}\)\>0andv1,v^1v\_\{1\},\\widehat\{v\}\_\{1\}are unit\-norm top eigenvectors, then the following inequality holds

sin⁡∠​\(v^1,v1\)≤‖B^λ−Bλ‖2γλ\.\\sin\\angle\(\\widehat\{v\}\_\{1\},v\_\{1\}\)\\leq\\frac\{\\\|\\widehat\{B\}\_\{\\lambda\}\-B\_\{\\lambda\}\\\|\_\{2\}\}\{\\gamma\_\{\\lambda\}\}\.The proof is hence complete\.

### B\.5Proof of Theorem[5\.1](https://arxiv.org/html/2606.04180#S5.Thmtheorem1)

We first prove the Frobenius concentration bound \([24](https://arxiv.org/html/2606.04180#S5.E24)\) by applying a Hilbert\-space Hoeffding inequality directly to the random matrices \(viewed as vectors under the Frobenius norm\)\. We then derive the eigenspace bound \([25](https://arxiv.org/html/2606.04180#S5.E25)\) via Davis–Kahan\.

Note that, by assumption,K~=1r​∑ℓ=1rK\(ℓ\)\\widetilde\{K\}=\\frac\{1\}\{r\}\\sum\_\{\\ell=1\}^\{r\}K^\{\(\\ell\)\}, where the matricesK\(ℓ\)K^\{\(\\ell\)\}are i\.i\.d\.,𝔼​\[K\(ℓ\)\]=K\\mathbb\{E\}\[K^\{\(\\ell\)\}\]=K, and\|Ki​j\(ℓ\)\|≤1\|K^\{\(\\ell\)\}\_\{ij\}\|\\leq 1almost surely for alli,ji,j\. Define the normalized matrices

A\(ℓ\):=1n​K\(ℓ\),A:=1n​K,A~:=1n​K~=1r​∑ℓ=1rA\(ℓ\)\.A^\{\(\\ell\)\}:=\\frac\{1\}\{n\}K^\{\(\\ell\)\},\\qquad A:=\\frac\{1\}\{n\}K,\\qquad\\widetilde\{A\}:=\\frac\{1\}\{n\}\\widetilde\{K\}=\\frac\{1\}\{r\}\\sum\_\{\\ell=1\}^\{r\}A^\{\(\\ell\)\}\.Then𝔼​\[A\(ℓ\)\]=A\\mathbb\{E\}\[A^\{\(\\ell\)\}\]=A, and

A~−A=1r​∑ℓ=1r\(A\(ℓ\)−A\)\.\\widetilde\{A\}\-A=\\frac\{1\}\{r\}\\sum\_\{\\ell=1\}^\{r\}\\bigl\(A^\{\(\\ell\)\}\-A\\bigr\)\.We work in the Hilbert space\(ℝn×n,⟨⋅,⋅⟩F\)\(\\mathbb\{R\}^\{n\\times n\},\\langle\\cdot,\\cdot\\rangle\_\{F\}\)where‖M‖=‖M‖F\\\|M\\\|=\\\|M\\\|\_\{F\}\. We letXℓ:=A\(ℓ\)−AX\_\{\\ell\}:=A^\{\(\\ell\)\}\-A\. Then,\{Xℓ\}ℓ=1r\\\{X\_\{\\ell\}\\\}\_\{\\ell=1\}^\{r\}are i\.i\.d\. random elements of this Hilbert space with𝔼​\[Xℓ\]=0\\mathbb\{E\}\[X\_\{\\ell\}\]=0\. Next, we derive applicable upper\-bounds on‖Xℓ‖F\\\|X\_\{\\ell\}\\\|\_\{F\}that hold with provable probability\. Since\|Ki​j\(ℓ\)\|≤1\|K^\{\(\\ell\)\}\_\{ij\}\|\\leq 1holds deterministically, we have the following for every indexℓ\\ell,

‖A\(ℓ\)‖F2=∑i=1n∑j=1n\(Ki​j\(ℓ\)n\)2≤∑i=1n∑j=1n1n2=1,\\\|A^\{\(\\ell\)\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\\Bigl\(\\frac\{K^\{\(\\ell\)\}\_\{ij\}\}\{n\}\\Bigr\)^\{2\}\\leq\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\\frac\{1\}\{n^\{2\}\}=1,and hence‖A\(ℓ\)‖F≤1\\\|A^\{\(\\ell\)\}\\\|\_\{F\}\\leq 1almost surely\. Similarly, using\|Ki​j\|≤1\|K\_\{ij\}\|\\leq 1for alli,ji,j,

‖A‖F2=∑i=1n∑j=1n\(Ki​jn\)2≤∑i=1n∑j=1n1n2=1,\\\|A\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\\Bigl\(\\frac\{K\_\{ij\}\}\{n\}\\Bigr\)^\{2\}\\leq\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\\frac\{1\}\{n^\{2\}\}=1,Therefore,‖A‖F≤1\\\|A\\\|\_\{F\}\\leq 1holds\. Hence, using the triangle inequality, we can show

‖Xℓ‖F=‖A\(ℓ\)−A‖F≤‖A\(ℓ\)‖F\+‖A‖F≤2\\\|X\_\{\\ell\}\\\|\_\{F\}=\\\|A^\{\(\\ell\)\}\-A\\\|\_\{F\}\\leq\\\|A^\{\(\\ell\)\}\\\|\_\{F\}\+\\\|A\\\|\_\{F\}\\leq 2Thus, the centered summands are uniformly bounded in norm byL=2L=2\.

We now apply the Hoeffding inequality for random vectors\(Sutherlandet al\.,[2018](https://arxiv.org/html/2606.04180#bib.bib13)\)to the i\.i\.d\. sequence\{Xℓ\}ℓ=1r\\\{X\_\{\\ell\}\\\}\_\{\\ell=1\}^\{r\}, withℓ2\\ell\_\{2\}\-norm upper\-boundL=2L=2\. This shows that for everyδ\>0\\delta\>0, the following holds with probability at least1−δ1\-\\delta:

‖1r​∑ℓ=1rXℓ‖F≤2r​\(1\+2​log⁡1δ\)\.\\Bigl\\\|\\frac\{1\}\{r\}\\sum\_\{\\ell=1\}^\{r\}X\_\{\\ell\}\\Bigr\\\|\_\{F\}\\leq\\frac\{2\}\{\\sqrt\{r\}\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)\.Recalling that1r​∑ℓ=1rXℓ=A~−A=1n​\(K~−K\)\\frac\{1\}\{r\}\\sum\_\{\\ell=1\}^\{r\}X\_\{\\ell\}=\\widetilde\{A\}\-A=\\frac\{1\}\{n\}\(\\widetilde\{K\}\-K\), we obtain

1n​‖K~−K‖F=‖A~−A‖F≤2r​\(1\+2​log⁡1δ\)\.\\frac\{1\}\{n\}\\\|\\widetilde\{K\}\-K\\\|\_\{F\}=\\\|\\widetilde\{A\}\-A\\\|\_\{F\}\\leq\\frac\{2\}\{\\sqrt\{r\}\}\\Bigl\(1\+\\sqrt\{2\\log\\tfrac\{1\}\{\\delta\}\}\\Bigr\)\.Multiplying both sides bynnyields the claimed Frobenius bound \([24](https://arxiv.org/html/2606.04180#S5.E24)\)\.

Subsequently, note that bothKKandK~\\widetilde\{K\}are symmetric matrices\. LetUUandU~\\widetilde\{U\}be the top\-qqeigenspaces ofKKandK~\\widetilde\{K\}\(represented byn×qn\\times qmatrices with orthonormal columns\), and assume the eigengapΔq​\(K\)=λq​\(K\)−λq\+1​\(K\)\>0\\Delta\_\{q\}\(K\)=\\lambda\_\{q\}\(K\)\-\\lambda\_\{q\+1\}\(K\)\>0\. A standard Davis–Kahansin⁡Θ\\sin\\Thetabound gives

‖sin⁡Θ​\(U~,U\)‖F≤‖K~−K‖2Δq​\(K\)\.\\big\\\|\\sin\\Theta\(\\widetilde\{U\},U\)\\big\\\|\_\{F\}\\leq\\frac\{\\big\\\|\\widetilde\{K\}\-K\\big\\\|\_\{2\}\}\{\\Delta\_\{q\}\(K\)\}\.Finally, for any matrixEE,‖E‖2≤‖E‖F\\big\\\|E\\big\\\|\_\{2\}\\leq\\big\\\|E\\big\\\|\_\{F\}\. Applying this withE=K~−KE=\\widetilde\{K\}\-Kyields

‖sin⁡Θ​\(U~,U\)‖F≤‖K~−K‖FΔq​\(K\)\.\\big\\\|\\sin\\Theta\(\\widetilde\{U\},U\)\\big\\\|\_\{F\}\\leq\\frac\{\\big\\\|\\widetilde\{K\}\-K\\big\\\|\_\{F\}\}\{\\Delta\_\{q\}\(K\)\}\.The above inequality completes the proof\.

## Appendix CAdditional Numerical Results

### C\.1Experiment Details

All experiments are conducted in the covariance\-operator formulation of KODA\. To ensure numerical stability and efficiency, the associated generalized eigenvalue problems are solved via Cholesky decomposition of the constrained covariance operator, following standard practice in kernel\-based spectral methods\. Throughout all experiments, we adopt a Gaussian \(RBF\) kernel without other specifications\. To enable scalable computation for both unimodal and multi\-modal settings, kernel features are approximated using Random Fourier Features \(RFF\)\. Unless otherwise specified, we use3,0003\{,\}000random Fourier features to approximate each Gaussian kernel\. The kernel bandwidthσ\\sigmais selected following common practice in the representation comparison literature\. We adopt the same bandwidth selection strategy as in prior work\(Zhanget al\.,[2024](https://arxiv.org/html/2606.04180#bib.bib42)\)\(Jalaliet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib9)\)\. Specifically, for each pair of embeddings under comparison, we tune the kernel bandwidths such that the leading eigenvalues of the resulting kernel matrices are of comparable magnitude across models, ensuring that neither embedding dominates the optimization due to scale differences\. For the quadratic constraint in KODA, the thresholdϵ\\epsiloncontrols the degree of weak clustering enforced under the constrained embedding\. Unless otherwise stated, we setϵ\\epsilonto the0\.50\.5quantile of the eigenvalues of the constrained model\. All experiments are performed on two NVIDIA RTX 4090 GPUs\.

### C\.2Sanity Check of Unimodal Comparison

Visualizing Pairwise Discrepancies via Kernel Difference Heatmaps\.[Figure5](https://arxiv.org/html/2606.04180#A3.F5)shows normalized RBF kernel similarities induced by CLIP and DINOv2 on ImageWoof \(ImageNet\-1k dog breeds\), together with their difference matrix\. The difference heatmap exhibits structured, high\-magnitude regions, where darker values indicate stronger mismatch between the two embeddings under the same similarity metric\. These mismatches concentrate on specific breed\-level relationships, suggesting that the discrepancies arise along meaningful semantic directions rather than random fluctuations\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x5.png)

Figure 5:Kernel similarity heatmaps induced by CLIP and DINOv2 on the ImageNet\-1k dog breeds, together with their difference\. \(scaled by 100 for better visualization\.\)Identified Discrepancy Directions Consistent with Kernel Difference Structures\.Based on the above normalized RBF kernel difference between the two embeddings, we identify the dog\-breed categories associated with the largest aggregated pairwise mismatches as a ground\-truth reference of semantic discrepancy\. We then apply KODA to the same dataset without using any label information to discover discrepancy directions between the two embeddings\. For each discovered direction, we select the top\-6 images for visualization\. As shown in[Figure6](https://arxiv.org/html/2606.04180#A3.F6), the top discrepancy directions recovered by KODA correspond closely to the most mismatched dog\-breed categories identified from the kernel difference matrix\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x6.png)

Figure 6:Consistency between dominant kernel mismatches \(ground truth\) and discrepancy directions identified by KODA on ImageNet dog breeds\.Left:the kernel difference matrix between DINOv2 and CLIP computed using normalized RBF kernels\.Middle:the top\-3 ground\-truth dog breeds associated with the largest aggregated mismatch scores in the difference matrix, together with representative images\.Right:representative samples from the top\-3 discrepancy directions identified by KODA without using label information\.
### C\.3Additional Results on Unimodal Comparison

We provide additional qualitative results for unimodal embedding comparison to complement the main experiments\. In particular, we analyze the dominant discrepancy directions discovered by KODA between DINOv2 and CLIP embeddings on AFHQ\(Choiet al\.,[2020](https://arxiv.org/html/2606.04180#bib.bib14)\)dataset\. We consider both asymmetric comparison settings: \(i\) directions that are weakly clustered under CLIP while being strongly grouped under DINOv2, and \(ii\) directions that are weakly clustered under DINOv2 while being strongly grouped under CLIP\. For each setting, we visualize the top discrepancy components obtained from KODA by inspecting the samples associated with the leading directions, as shown in the[Figure7](https://arxiv.org/html/2606.04180#A3.F7)and[Figure8](https://arxiv.org/html/2606.04180#A3.F8)\. Also, we compare directions identified by KODA with those from the SPEC\(Jalaliet al\.,[2025a](https://arxiv.org/html/2606.04180#bib.bib9)\)baseline, using the same settings\. We quantify the strength of a discrepancy directionxxusing the generalized Rayleigh quotientx⊤​K1​xx⊤​K2​x\\frac\{x^\{\\top\}K\_\{1\}x\}\{x^\{\\top\}K\_\{2\}x\}, whereK1K\_\{1\}andK2K\_\{2\}are normalized RBF kernel matrices induced by the two embeddings\. Sincex⊤​K​xx^\{\\top\}Kxmeasures how strongly directionxxis expressed under kernelKK, larger quotient values indicate stronger directional asymmetry, i\.e\., directions emphasized byK1K\_\{1\}but suppressed byK2K\_\{2\}\.[Figure9](https://arxiv.org/html/2606.04180#A3.F9)reports the quotient values of KODA’s Top\-1 direction and the averages over Top\-3 and Top\-5 directions across different quantiles, together with SPEC’s Top\-1 direction\. Across all constraint levels, KODA consistently achieves substantially larger quotient values; notably, atq=0\.1q=0\.1, KODA’s Top\-1 direction is about20×20\\timesstronger than SPEC, and even the Top\-5 average remains clearly above SPEC throughout\. These additional results further demonstrate the ability of KODA to disentangle directional discrepancies that depend on the choice of reference embedding, even in unimodal settings\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x7.png)

Figure 7:Top\-10 DINOv2 dominant directions relative to CLIP on the AFHQ dataset identified by KODA, visualized via representative samples for each direction\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x8.png)

Figure 8:Top\-10 CLIP dominant directions relative to DINOv2 on the AFHQ dataset identified by KODA, visualized via representative samples for each direction\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x9.png)

Figure 9:Left: Visualization of the top\-5 mismatch directions of DINOv2 and CLIP on the AFHQ dataset discovered by KODA \(ours\) and SPEC \(baseline\), respectively\. Right: Generalized Rayleigh quotientx⊤​K1​xx⊤​K2​x\\frac\{x^\{\\top\}K\_\{1\}x\}\{x^\{\\top\}K\_\{2\}x\}w\.r\.t\. the constraint onK2K\_\{2\}\. \(The quotient can be interpreted as a multiplicative measure of how strongly a given direction is represented in DINOv2 relative to CLIP\.\)
### C\.4Additional Results on Multimodal Comparison

We further provide additional results for multimodal embedding comparison on the MS\-COCO dataset\. In this setting, we analyze discrepancy directions discovered by KODA across a diverse set of vision–language models, including CLIP, OpenCLIP, BLIP, SigLIP, and SigLIP2\. For each pair of multimodal embeddings, we construct joint image–text kernels and apply KODA to identify dominant discrepancy directions under asymmetric constraints\. We visualize the samples associated with the top discrepancy components by inspecting the image–text pairs corresponding to the leading directions, as shown in the[Figure12](https://arxiv.org/html/2606.04180#A3.F12)\-[18](https://arxiv.org/html/2606.04180#A3.F18)\.

These visualizations highlight how different multimodal models organize paired image–text data in distinct ways, even when trained on similar objectives or datasets\. Across different model combinations, the dominant discrepancy directions correspond to different subsets of samples, reflecting variations in how visual and textual information is jointly encoded\. These additional results complement the main experiments by illustrating the generality of KODA across a wide range of multimodal embedding families\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x10.png)

Figure 10:Multimodal discrepancy analysis of SigLIP dominant directions relative to OpenCLIP on the MSCOCO dataset\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x11.png)

Figure 11:Multimodal discrepancy analysis of OpenCLIP dominant directions relative to SigLIP on the MSCOCO dataset\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x12.png)

Figure 12:Multimodal discrepancy analysis of BLIP dominant directions relative to CLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using BLIP and CLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x13.png)

Figure 13:Multimodal discrepancy analysis of CLIP dominant directions relative to BLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using CLIP and BLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x14.png)

Figure 14:Multimodal discrepancy analysis of OpenCLIP dominant directions relative to CLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using OpenCLIP and CLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x15.png)

Figure 15:Multimodal discrepancy analysis of CLIP dominant directions relative to OpenCLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using CLIP and OpenCLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x16.png)

Figure 16:Multimodal discrepancy analysis of SigLIP dominant directions relative to CLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using SigLIP and CLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x17.png)

Figure 17:Multimodal discrepancy analysis of SigLIP2 dominant directions relative to CLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using SigLIP2 and CLIP embeddings respectively\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x18.png)

Figure 18:Multimodal discrepancy analysis of SigLIP dominant directions relative to OpenCLIP on the MSCOCO dataset\.Top:Representative image–caption pairs corresponding to the Top\-3 discrepancy directions identified by KODA\.Bottom:t\-SNE visualization of Top\-10 discrepancy directions using SigLIP and OpenCLIP embeddings respectively\.
### C\.5Ablation Study

We conduct a set of ablation studies to examine the sensitivity of KODA to key design choices, including the number of random Fourier features, the size of the reference sample set, and the choice of kernel function\. All ablation experiments are conducted using the same experimental protocol as in the main results, with only the specified factor varied while keeping other settings fixed\.

Number of Random Fourier Features\.We examine the effect of the number of random Fourier features used to approximate the Gaussian kernel by varying the feature dimensionalityr∈\{500,1000,2000,3000\}r\\in\\\{500,1000,2000,3000\\\}\. For each setting, we visualize the dominant discrepancy directions discovered by KODA in[Figure19](https://arxiv.org/html/2606.04180#A3.F19)\. As the number of random features increases, the discovered discrepancy directions become progressively more coherent and visually well\-separated\. In particular, higher\-dimensional approximations lead to cleaner and more stable grouping patterns, while lower\-dimensional approximations exhibit increased noise in the dominant directions\. These results indicate that sufficiently rich random feature approximations are beneficial for stable discrepancy discovery, and motivate our choice ofr=3000r=3000in the main experiments\.

Kernel Function\.We investigate the effect of the kernel function by comparing Gaussian \(RBF\) kernels with cosine similarity kernels\. For each kernel choice, we apply KODA using the same constraint setting and visualize the dominant discrepancy directions\. As[Figure20](https://arxiv.org/html/2606.04180#A3.F20)shows, different kernel functions lead to different discrepancy patterns, reflecting the distinct geometric properties emphasized by each kernel\. In particular, Gaussian kernels capture local neighborhood structure based on Euclidean distance, whereas cosine similarity kernels emphasize angular relationships between representations\. As a result, the dominant discrepancy directions discovered under different kernels correspond to different groupings of samples\.

Reference Sample Size\.We study the effect of the reference sample size by varying the number of samplesn∈\{2000,4000,8000,16000\}n\\in\\\{2000,4000,8000,16000\\\}while keeping all other settings fixed\. For each choice ofnn, we apply KODA and visualize the dominant discrepancy directions in[Figure21](https://arxiv.org/html/2606.04180#A3.F21)\. As the sample size increases, the discovered discrepancy directions become increasingly stable and consistent\. In particular, whenn=16,000n=16\{,\}000, the resulting discrepancy components exhibit highly stable grouping patterns, indicating that sufficient reference coverage is important for reliable discrepancy discovery\. At smaller sample sizes, the overall structure of the discrepancy directions is preserved, albeit with increased variability in the visualizations\. Notably, performing spectral decomposition directly on kernel matrices of size exceeding10,000×10,00010\{,\}000\\times 10\{,\}000is often impractical on modern GPUs due to memory and computational constraints\. By operating in the covariance space induced by random feature representations, KODA reduces the effective dimensionality of the spectral problem to the feature dimension \(e\.g\.,6,0006\{,\}000in our experiments\), making eigen\-decomposition feasible even when the number of reference samples is large\. This formulation enables stable discrepancy analysis at larger sample sizes without requiring explicit construction or decomposition of full kernel matrices\.

![Refer to caption](https://arxiv.org/html/2606.04180v1/x19.png)

Figure 19:Multimodal discrepancy analysis of SigLIP dominant directions relative to OpenCLIP on the MSCOCO dataset under different number of joint random fourier features\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x20.png)

Figure 20:Multimodal discrepancy analysis of OpenCLIP dominant directions relative to CLIP on the MSCOCO dataset under gaussian kernel function or cosine kernel function\.![Refer to caption](https://arxiv.org/html/2606.04180v1/x21.png)

Figure 21:Multimodal discrepancy analysis of SigLIP dominant directions relative to OpenCLIP on the MSCOCO dataset under different number of sample size\.