用于对比学习中语义几何的锚点散度

# Anchor Divergence for Semantic Geometry in Contrastive Learning
Source: [https://arxiv.org/html/2610.06919](https://arxiv.org/html/2610.06919)
Kiho ParkAffiliation:University of ChicagoVictor VeitchAffiliation:University of Chicago

###### Abstract

This paper concerns how semantic context determines geometry in learned vector representations\. Similarity is typically measured using cosine similarity, which provides a single fixed geometry\. Semantic similarity, however, is inherently context dependent: two images may be similar because they depict the same object, share a visual style, or are relevant to the same clinical finding\. We show that contrastive representations naturally encompass a family of geometries that can be specialized to particular semantic structure\. The key idea is to use an interplay between contrastive learning, exponential families, and information geometry to establish a correspondence between probability distributions over “anchors” and Bregman geometries on the representation space\. We use this correspondence to define*Anchor Divergences*, a method for specifying context\-specific semantic geometries on fixed representations\. Under this correspondence, modeling the anchor distribution models the geometry itself\. Experiments on retrieval show that anchor divergences provide an effective and efficient way to specify context\-specific semantic similarity\. Code is available at[github\.com/sky1712/Anchor\-Divergence](https://github.com/sky1712/Anchor-Divergence)\.

## 1Introduction

Modern AI systems encode semantic structure in dense vector representations\. This paper asks how similarity between such representations should be measured\. We focus on contrastive models such as CLIP and its successors\[[Rad\+21](https://arxiv.org/html/2610.06919#bib.bibx25)\], which are trained to score matched query–key pairs above mismatched ones using cosine\-based softmax objectives\. It is therefore natural to use cosine similarity as the geometry of the learned space\. But this creates a basic mismatch: cosine provides a single fixed notion of similarity, while semantic similarity is inherently context dependent\.

For concreteness, we focus on image similarity in vision–language models, and study how semantic similarity should be defined between pairs of images\.[Figure1](https://arxiv.org/html/2610.06919#S1.F1)illustrates the fundamental limitation of cosine similarity in this setting\. A query image may share different semantic attributes with different database images, while cosine implicitly fixes how these sources of variation contribute to similarity\. The relevant notion, however, depends on context: we may care about the same dog breed, background, or outfit\. Understanding how to specify such context\-specific similarity is important both foundationally—the question of how dense representations encode semantics is the heart of \(mechanistic\) interpretability—and practically\. For example, when retrieving chest X\-rays similar to a query image, we may want similarity to reflect specific pathologies rather than unrelated patient characteristics or acquisition artifacts; see[Figure2](https://arxiv.org/html/2610.06919#S3.F2)\. A fixed cosine geometry has no reason to privilege the semantic aspect relevant to the task\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/query_041_breed-siberian_husky__background-snowy_mountains__clothes-yellow_raincoat.png)Figure 1:Anchor divergences induce context\-specific neighborhoods for the same query\.Cosine similarity gives a single fixed ranking\. In contrast, changing the semantic anchors changes which aspect of the query defines its neighborhood\. For example, breed anchors retrieve other Huskies while allowing background and clothing to vary; background and clothing anchors behave analogously\. The underlying image representations remain fixed throughout\.The main contribution of this paper is to show that the contrastive structure naturally defines a*family*of semantic geometries on the image representation space, each induced by a probability distribution over semantically relevant “anchors\.” This correspondence leads to*anchor divergences*, a method for specifying context\-specific semantic geometries on fixed representations; see[Figure1](https://arxiv.org/html/2610.06919#S1.F1)\.

Informally, the starting idea is to specify the semantics of interest using a set of text anchors\. For example, in the chest X\-ray setting, the anchors might consist of clinical descriptions of pathologies—the variation in the text includes only the aspects we care about\. Relative to any such anchor set, the contrastive model uses the same softmax structure that appears during training to map each image to a distribution over the anchors, based on their image–text similarities\. We can then compare two images through the distributions they induce over the anchors\. In this way, the chosen anchors determine which semantic differences are emphasized by the comparison\.

In principle, we could define context\-specific similarity directly using a finite anchor set\. However, this ties the geometry to a particular collection of anchors: it may reflect idiosyncrasies of that finite set, and the cost of representing the geometry grows with the number of anchors\. The main technical insight is to move from identifying the geometry with a finite set of text anchors to identifying it with a probability distribution from which the anchors are drawn i\.i\.d\. Using ideas from information geometry, we show that each such distribution corresponds to a Bregman divergence defined on the original image embeddings\. This shifts the object specifying semantic geometry from a finite anchor set to an underlying semantic distribution, which can in turn be modeled efficiently\.

The development of the paper is:

1. 1\.First, we show that the KL divergence between softmax distributions over a finite set of semantic anchors has a natural limit as a Bregman divergence on image representations\. This reveals a direct role of information geometry in organizing semantic similarity in contrastive representations, and leads to*anchor divergences*—a principled method for specifying context\-specific semantic geometries\.
2. 2\.A key consequence of the Anchor Divergence construction is that the anchor distribution determines the induced Bregman geometry\. Drawing on classical connections between exponential families and Bregman geometry\[[Efr78](https://arxiv.org/html/2610.06919#bib.bibx10),[Ama82](https://arxiv.org/html/2610.06919#bib.bibx2),[WJ08](https://arxiv.org/html/2610.06919#bib.bibx34),[AC10](https://arxiv.org/html/2610.06919#bib.bibx3)\], we use the observed anchors to fit a parametric model of the underlying population anchor distribution, whose fitted distribution induces an explicit Bregman divergence\. This has two practical benefits: once the model is fit, the cost of representing and evaluating the geometry need not scale with the number of observed anchors, and the parametric structure can provide useful approximations when only a limited number of anchors are available\.
3. 3\.As a further consequence of modeling the anchor distribution, we show how broad semantic structure can be adapted to a specific target using an empirical\-Bayes\-type construction\. The idea is to first learn a broad anchor distribution—and hence a broad semantic geometry—from a large collection of anchors that are loosely related to the target, e\.g\., all text about diseases, and then refine it using a small set of targeted anchors\. Empirically, we find that this can reduce task\-specific anchor requirements by a full order of magnitude\.
4. 4\.Finally, experiments test the consequences of the framework\. We show that the anchor distribution controls which semantic factors govern similarity; parametric models are useful when anchors are limited, while the non\-parametric softmax improves as anchor coverage grows; and broad semantic structure can be adapted using few target anchors\. Across these settings, the resulting geometries improve over task\-agnostic cosine similarity and task\-specific heuristic approaches\.

## 2Anchor Divergence

We begin with a finite set of text anchors�=\{,1…,\}N\\Gamma=\\\{\{\}\_\{1\},\\dots,\{\}\_\{N\}\\\}chosen to describe the semantics of interest\. For example, in a pathology\-specific geometry, the anchors may be clinical descriptions of different findings\. Given an image embedding�\\lambda, the contrastive softmax induces a distribution over these anchors,P�\(\|k�\)∝exp\(�⊤/k�\),\\mdmathbb\{P\}\_\{\\Gamma\}\(\{\}\_\{k\}\\penalty\\ \|\\penalty\\ \\lambda\)\\propto\\exp\(\\lambda^\{\\top\}\{\}\_\{k\}/\\tau\),where�\>0\\tau\>0is the temperature\. A natural way to compare two image embeddings is then by the KL divergence between the distributions they induce:

dSM,�\(�1∥�2\):=KL\(P�\(⋅\|�1\)∥P�\(⋅\|�2\)\)\.d\_\{\\mathrm\{SM\},\\Gamma\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\):=\\mathrm\{KL\}\\\!\\left\(\\mdmathbb\{P\}\_\{\\Gamma\}\(\\cdot\\penalty\\ \|\\penalty\\ \\lambda\_\{1\}\)\\,\\\|\\,\\mdmathbb\{P\}\_\{\\Gamma\}\(\\cdot\\penalty\\ \|\\penalty\\ \\lambda\_\{2\}\)\\right\)\.We will refer to this as the*softmax anchor divergence*\.

Specifying a divergence via a finite set of anchors is appealingly simple\. However, we may worry that the procedure may be overly sensitive to the specific choice of anchors\. For example, if swapping out a single clinical note substantively altered the geometry of the pathology\-specific anchor divergence, it would suggest that we have not captured a meaningful notion of pathology\-specific geometry\. This motivates viewing the observed anchors as samples from an underlying distribution over semantically relevant anchors\. In this view, the distribution, rather than any particular finite realization, specifies the semantic context\. We now see that such a distribution induces a well\-defined population geometry on the image embedding space\.

To state the result, we recall two standard definitions\.

###### Definition 1\.

For differentiable, strictly convexff, the*Bregman divergence*is

Bf\(�1∥�2\):=f\(�1\)−f\(�2\)−∇f\(�2\)⊤\(�1−�2\)\.B\_\{f\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\):=f\(\\eta\_\{1\}\)\-f\(\\eta\_\{2\}\)\-\\nabla f\(\\eta\_\{2\}\)^\{\\top\}\(\\eta\_\{1\}\-\\eta\_\{2\}\)\.

###### Definition 2\.

For a distributionqqonRd\\mdmathbb R^\{d\}, its*cumulant generating function*\(CGF\) is

�q\(�\):=log∫exp\(�⊤\)q\(d\)\.\\kappa\_\{q\}\(\\eta\):=\\log\\int\\exp\(\\eta^\{\\top\}\\gamma\)\\,q\(d\\gamma\)\.

The next result identifies the corresponding population geometry\.

###### Theorem 3\(Softmax anchor divergence converges to a population Bregman divergence\)\.

Let�N=\{,1…,\}N\\Gamma\_\{N\}=\\\{\{\}\_\{1\},\\dots,\{\}\_\{N\}\\\}be sampled i\.i\.d\. from an anchor distributionqqonSd−1\\mdmathbb S^\{d\-1\}\. Then, asN→∞N\\to\\infty,

sup�1,�2∈Sd−1\|dSM,�N\(�1∥�2\)−B�q\(�2�∥�1�\)\|→a\.s\.0\.\\displaystyle\\sup\_\{\\lambda\_\{1\},\\lambda\_\{2\}\\in\\mdmathbb S^\{d\-1\}\}\\left\|d\_\{\\mathrm\{SM\},\\Gamma\_\{N\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)\-B\_\{\\kappa\_\{q\}\}\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\)\\right\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.\(2\.1\)

The proof is deferred to[AppendixA](https://arxiv.org/html/2610.06919#A1)\. The key observation is that the contrastive softmax is an exponential tilt of the empirical measure over anchors\. The resulting natural exponential family links KL divergence to the Bregman geometry generated by�q\\kappa\_\{q\}\.

Thus, the simple idea of comparing images through their distributions over semantic anchors yields a population Bregman geometry determined by the anchor distribution\. This motivates our central definition\.

###### Definition 4\.

For a distributionqqonRd\\mdmathbb R^\{d\}, the*anchor divergence*is

dq\(�1∥�2\):=B�q\(�2�∥�1�\)\.\\displaystyle d\_\{q\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\):=B\_\{\\kappa\_\{q\}\}\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\)\.\(2\.2\)

Under this definition, the anchor distributionqqspecifies the semantic context and thereby determines the geometry on the image embedding space\. The softmax anchor divergence is the special case induced by the empirical distribution\. This distributional view will allow us to model and adapt the semantic geometry in the following sections\.

## 3Modeling Structured Semantic Geometries

We have arrived at anchor divergences as Bregman divergences induced by an anchor*distribution*qqthat represents the semantics of interest\. In practice, however, we observe only a finite set of anchors\. We therefore turn to modeling the underlying distribution from these observations, and hence the induced semantic geometry\.

The simplest approach is to use the empirical measure on the anchor set\. It is straightforward to verify that the anchor divergence associated with this empirical distribution is exactly the softmax anchor divergence defined earlier\. As established by[Theorem3](https://arxiv.org/html/2610.06919#Thmtheorem3)this will converge to the target population geometry as we get more and more anchors\. With limited anchors, however, the finite anchor set may poorly represent the underlying semantics\.

Sinceqqdetermines the induced geometry, we can impose structure on the geometry by modelingqq\. The idea is to specify a parametric modelq�q\_\{\\theta\}for the anchor distribution\. For example, we may takeq�q\_\{\\theta\}to be a multivariate Gaussian distribution with�\\thetaits mean and covariance\. We then fit�^\\hat\{\\theta\}to the specified anchors, say by maximum likelihood\. The fitted distributionq�^q\_\{\\hat\{\\theta\}\}then induces the corresponding anchor divergence through[Definition4](https://arxiv.org/html/2610.06919#Thmtheorem4)\. Thus, different anchor\-distribution assumptions induce different structured geometries in image embedding space\.

This reflects the usual tradeoff of parametric modeling\. Simpler models impose stronger assumptions, which can help when anchors only partially specify the target semantics, but may miss the target geometry if those assumptions are too restrictive, even with many anchors\. Moreover, once fit, the divergence is evaluated through the fitted distribution rather than allNNobserved anchors, so its evaluation cost need not scale with the anchor\-set size; we examine this empirically in[SectionF\.4](https://arxiv.org/html/2610.06919#A6.SS4)\.

Table 1:Modeling the anchor distribution induces different semantic geometries\.Each column specifies an anchor\-distribution model and its CGF\. The empirical model places uniform mass on the observed anchors; for the Gaussian model,�^\\hat\{\\mu\}and�^\\hat\{\\Sigma\}are the sample mean and covariance; for the Gaussian mixture,\{\(�^m,�^m,�^m\)\}m=1M\\\{\(\\hat\{\\pi\}\_\{m\},\\hat\{\\mu\}\_\{m\},\\hat\{\\Sigma\}\_\{m\}\)\\\}\_\{m=1\}^\{M\}are fit by expectation–maximization\.[Table1](https://arxiv.org/html/2610.06919#S3.T1)illustrates this perspective for three anchor models\. The empirical distribution recovers softmax anchor divergence\. The Gaussian model is the simplest parametric alternative: fitting its mean and covariance yields a quadratic CGF, and hence an elliptical geometry\. The resulting geometry emphasizes directions along which the modeled anchor distribution varies\. A Gaussian mixture allows richer structure: its CGF is a log\-sum\-exp of quadratic potentials, allowing the induced geometry to reflect multiple semantic modes rather than a single global covariance structure\.

##### Bregman divergence in a projected anchor space\.

We also consider mildly generalizing from Bregman divergences on the contrastive embedding space to Bregman divergences on a subspace of the embedding space\. This is motivated by evidence that semantic structure is often encoded in low\-dimensional subspaces\[[Old\+23](https://arxiv.org/html/2610.06919#bib.bibx21),[Bha\+24](https://arxiv.org/html/2610.06919#bib.bibx7),[Eng\+25](https://arxiv.org/html/2610.06919#bib.bibx11),[Par\+25](https://arxiv.org/html/2610.06919#bib.bibx22)\]\. We learn a projection, optionally followed by normalization, onto a semantically relevant subspace from the anchor set and define the Anchor Divergence on the projected space\. In[AppendixB](https://arxiv.org/html/2610.06919#A2), we show that this corresponds to a latent\-variable model of the anchor distribution and yields a procedure for estimating the projector\. We refer to the resulting divergences as*latent Gaussian*and*latent spherical*anchor divergences\. Operationally, both project onto the top\-kkeigenvectors of the empirical anchor covariance; the latent Gaussian divergence then uses Euclidean distance, while the latent spherical divergence uses cosine distance\.111We note the latent spherical case yields the same projected cosine structure proposed in\[[HW26](https://arxiv.org/html/2610.06919#bib.bibx14)\]

The empirical and parametric constructions provide different ways of representing the anchor distribution, and hence different approximations to the same target semantic geometry\.[Figure2](https://arxiv.org/html/2610.06919#S3.F2)provides a qualitative illustration of the usefulness of the anchor divergence framework and its parametric approximations: both emphasize the pathology of interest over nuisance device\-related structure\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/chexpert_retrieval_comparison_grid.png)Figure 2:Anchor divergences can emphasize relevant semantics over nuisance variation\.The query X\-ray shows the target pathology, pleural effusion, as well as a support device\. Cosine similarity retrieves an image sharing device\-related structure but not the target pathology, whereas both softmax and \(Gaussian model\) parametric anchor divergences retrieve images with pleural effusion\.

## 4Adapting Semantic Geometries from Broad Anchors

Obtaining many target\-specific anchors can be cumbersome, while broader related anchors may be easier to obtain\. The distributional view enables an empirical\-Bayes approach: broad anchors provide prior information about the parameters of the anchor distribution, which is then updated using scarce target anchors\. For example, in pathology\-specific retrieval, we may have abundant general clinical text but only a few precise descriptions of the target pathology\. The goal is to specialize broad semantic structure using a small number of pathology\-specific anchors\.

Let�broad\\Gamma\_\{\\mathrm\{broad\}\}and�target\\Gamma\_\{\\mathrm\{target\}\}denote the broad and target anchor sets, respectively\. Let\{q�:�∈�\}\\\{q\_\{\\theta\}:\\theta\\in\\Theta\\\}be a parametric anchor model with priorp⁡\(�\|\)p\(\\theta\\penalty\\ \|\\penalty\\ \\Omega\)\. We adapt the anchor distribution in three steps:

1. 1\.fitq�q\_\{\\theta\}to�broad\\Gamma\_\{\\mathrm\{broad\}\}and use the fit to set the prior hyperparameters\\Omega;
2. 2\.update this prior using�target\\Gamma\_\{\\mathrm\{target\}\}via a model\-specific posterior or posterior\-like update;
3. 3\.summarize the update by�^\\hat\{\\theta\}, such as a posterior mean or MAP estimate\.

The adapted distributionq�^q\_\{\\hat\{\\theta\}\}induces the corresponding semantic geometry through its anchor divergence, giving the following definition\.

###### Definition 5\.

Given�broad\\Gamma\_\{\\mathrm\{broad\}\}and�target\\Gamma\_\{\\mathrm\{target\}\}, letq�^q\_\{\\hat\{\\theta\}\}denote the adapted anchor distribution obtained by using�broad\\Gamma\_\{\\mathrm\{broad\}\}to define prior information and updating it with�target\\Gamma\_\{\\mathrm\{target\}\}\. The*posterior anchor divergence*is the anchor divergence induced byq�^q\_\{\\hat\{\\theta\}\}\.

All methods target the same object: a Bregman geometry induced by an anchor distribution\. They differ only in how that distribution is estimated\. Softmax anchor divergence uses the empirical distribution over target anchors; parametric anchor divergence fits a structured model to those anchors; posterior anchor divergence further regularizes this structured estimate using broad, related anchors as prior information\. When\|�target\|\|\\Gamma\_\{\\mathrm\{target\}\}\|is small, broad anchors stabilize the geometry; as\|�target\|\|\\Gamma\_\{\\mathrm\{target\}\}\|grows, target anchors increasingly determine it\. Model\-specific updates are in[AppendixD](https://arxiv.org/html/2610.06919#A4)\.

## 5Experiments

Our experiments are designed to test the geometric and statistical predictions of the framework\. We organize the experiments around the following questions:

1. 1\.Can Anchor Divergences effectively specialize geometry to different semantic factors?
2. 2\.With limited anchors, can parametric models usefully approximate the target geometry?
3. 3\.Does posterior anchor modeling enable semantic adaptation using few target anchors?

### 5\.1Implementation details

All natural\-image experiments\[[Rus\+15](https://arxiv.org/html/2610.06919#bib.bibx27),[Hen\+21](https://arxiv.org/html/2610.06919#bib.bibx13),[San\+16](https://arxiv.org/html/2610.06919#bib.bibx28),[Wah\+11](https://arxiv.org/html/2610.06919#bib.bibx33)\]use MetaCLIP\-2\[[Chu\+25](https://arxiv.org/html/2610.06919#bib.bibx8)\]\. Class\-name anchors use the prompt template “a photo of a \{class\}”, and style anchors use “a \{style\} rendition of an object”\. For the clinical experiments, we use CheXpert\[[Irv\+19](https://arxiv.org/html/2610.06919#bib.bibx15)\]with CheXzero\[[Tiu\+22](https://arxiv.org/html/2610.06919#bib.bibx31)\]\. Image and text embeddings are normalized to unitℓ2\\ell\_\{2\}norm, and we fix the inverse temperature1/�1/\\tauat the model\-provided logit scale\.

For asymmetric divergences, we rank keys using the symmetrized score

dsym\(�q,�k\)=12\[d\(�q∥�k\)\+d\(�k∥�q\)\]\.d\_\{\\mathrm\{sym\}\}\(\\lambda\_\{q\},\\lambda\_\{k\}\)=\\frac\{1\}\{2\}\\left\[d\(\\lambda\_\{q\}\\\|\\lambda\_\{k\}\)\+d\(\\lambda\_\{k\}\\\|\\lambda\_\{q\}\)\\right\]\.We report Area Under the Precision–Recall Curve \(AUPRC\), which is suitable when relevant keys are sparse and prevalence varies across semantic targets\. Relevance is defined by the semantic labels of each experiment\.

We compare against two reference geometries\. Cosine similarity provides the standard task\-agnostic geometry\. Image\-text inversion \(ITI\) provides a task\-specific heuristic baseline using a predict\-then\-retrieve procedure: it first matches the query image to the closest anchor and then ranks keys by cosine similarity to this anchor\. This follows the spirit of image inversion methods in\[[Bal\+23](https://arxiv.org/html/2610.06919#bib.bibx5),[Mis\+25](https://arxiv.org/html/2610.06919#bib.bibx18)\]\. See[AppendixE](https://arxiv.org/html/2610.06919#A5)for full implementation details\.

### 5\.2Anchor distributions select the semantic coordinate

We first isolate the effect of the anchor distribution with the embeddings fixed\.[Figure1](https://arxiv.org/html/2610.06919#S1.F1)gives a qualitative illustration\. We test this systematically: if the anchor distribution determines the semantic geometry, then changing it should change which semantic coordinate is preserved by retrieval\.

##### Clinical coordinate control\.

Table 2:Pathology\-specific anchors specialize clinical retrieval\.Condition\-wise AUPRC differences between softmax anchor divergence and each reference geometry; positive values favor anchor divergence\. Results are averaged over 10 independently generated anchor sets and 10 CheXzero checkpoints, with 95% intervals accounting for bootstrap and anchor\-generation variability\. Values\>0\.005\>0\.005are bold, values in\[−0\.005,0\.005\]\[\-0\.005,0\.005\]are gray, and values<−0\.005<\-0\.005are bold red\.For each CheXpert finding, we independently generate 10 sets of report\-style clinical text anchors from a fixed LLM prompt; see[SectionE\.6](https://arxiv.org/html/2610.06919#A5.SS6)\. We evaluate retrieval using the corresponding pathology label and average over anchor generations and the 10 released CheXzero checkpoints\.[Table2](https://arxiv.org/html/2610.06919#S5.T2)reports AUPRC differences relative to cosine similarity and ITI, with 95% intervals accounting for anchor\-generation and bootstrapping\. Anchor divergence improves over the reference geometries for most findings\. The largest gains occur for pleural effusion, cardiomegaly, edema, and support devices\. These results indicate that pathology\-specific anchors can shift retrieval toward the intended clinical coordinate beyond generic image\-space similarity\.

##### Object–style control\.

We next test whether varying the anchor distribution can interpolate between two semantic coordinates\. ImageNet\-R contains ImageNet object classes rendered in diverse styles, allowing retrieval to be evaluated by either object identity or style\. Letqobjq\_\{\\mathrm\{obj\}\}andqstyleq\_\{\\mathrm\{style\}\}denote the corresponding empirical anchor distributions, and mix them asqp=p​qobj\+\(1−p\)​qstyleq\_\{p\}=pq\_\{\\mathrm\{obj\}\}\+\(1\-p\)q\_\{\\mathrm\{style\}\}, withp∈\[0,1\]p\\in\[0,1\]\. We evaluate retrieval asppvaries, averaging over 10 random realizations of the anchor set\.[Figure3](https://arxiv.org/html/2610.06919#S5.F3)shows that increasing object\-anchor mass increases object retrieval while decreasing style retrieval\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/auprc_vs_p_combined.png)Figure 3:Anchor distributions select the semantic coordinate\.Left:Mixing object and style anchor distributions on ImageNet\-R shifts retrieval from preserving style to preserving object identity asppincreases\.Right:For a video\-game\-style space shuttle query, retrieved neighbors similarly shift from style\-matched to object\-matched images\.Together, these experiments show that the same fixed image embeddings support different semantic geometries solely through changes to the anchor distribution\.

### 5\.3Parametric modeling is useful with limited anchors

We next test whether parametric models of the anchor distribution provide useful approximations to the target geometry when only limited anchors are available\. Softmax uses the empirical anchor distribution directly, whereas parametric models impose additional structure\. We therefore expect parametric models to be most useful when anchors are limited, while the empirical construction should improve as anchor coverage grows\.

We first vary the number of ImageNet class anchors and evaluate retrieval on ImageNet\.[Figure4](https://arxiv.org/html/2610.06919#S5.F4)left shows the intuitive sample\-efficiency pattern: structured geometries recover useful semantic neighborhoods at moderate anchor budgets, while softmax improves with anchor coverage and eventually becomes strongest\. Similar trends hold across the other datasets; see[AppendixF](https://arxiv.org/html/2610.06919#A6)\.

We then consider the high\-coverage regime using full dataset\-matched anchor sets on ImageNet\-R, CUB\-200, and Sketchy\.[Figure4](https://arxiv.org/html/2610.06919#S5.F4)right shows that softmax anchor divergence is strongest in this setting, while parametric anchor divergences retain much of its gain over cosine similarity\. Together with the ImageNet results, this illustrates the modeling tradeoff from[Section3](https://arxiv.org/html/2610.06919#S3): parametric structure can be useful with limited anchors, while the empirical anchor distribution becomes increasingly effective as anchor coverage grows\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/aligned_anchors_combined_results.png)Figure 4:Parametric modeling is useful with limited anchors\.Left:On ImageNet, parametric anchor divergences are strongest in the red\-shaded limited\-anchor regime, while softmax improves with coverage and becomes strongest in the blue high\-coverage regime\.Right:With full dataset\-matched anchor sets on ImageNet\-R, CUB\-200, and Sketchy, softmax gives the strongest retrieval performance, while parametric anchor divergences retain much of its gain over cosine similarity\.
### 5\.4Posterior modeling adapts broad geometry with few target anchors

Finally, we test whether broad semantic structure can be adapted toward a target geometry using only a few task\-specific anchors\. This setting exposes a limitation of softmax anchor divergence: as a nonparametric estimator, it places mass only on observed target anchors, so with few anchors it can yield a poor approximation of the target geometry\. Posterior anchor models instead use broad, semantically related anchors as prior information and specialize this structure using target anchors\.

We evaluate this on ImageNet, using WordNet anchors as broad semantic structure and ImageNet class names as target\-specific anchors\. We vary the number of target anchors and compare target\-only softmax, latent spherical, and GMM anchor divergences with posterior latent spherical and posterior GMM models adapted from WordNet, alongside cosine similarity\. Results are averaged over five random seeds; model\-specific adaptation details are given in[AppendixD](https://arxiv.org/html/2610.06919#A4)\.

[Figure5](https://arxiv.org/html/2610.06919#S5.F5)shows that posterior anchor models are most useful when target anchors are scarce\. Broad anchors provide prior semantic structure, while target anchors specialize the geometry toward the ImageNet class coordinate\. As target\-anchor coverage grows, the gap to target\-only models narrows as the target anchors increasingly determine the geometry\.

##### Summary\.

The experiments support the main consequences of the framework\. Changing the anchor distribution changes which semantic factors govern similarity; parametric models provide useful approximations when anchors are limited, while the empirical softmax geometry improves with anchor coverage; and posterior anchor models adapt broad semantic structure using few task\-specific anchors\. Together, these results support anchor divergences as a flexible framework for specifying, modeling, and adapting context\-dependent semantic geometry in contrastive representation spaces\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/plot_auprc_subset_methods.png)Figure 5:Posterior anchor modeling adapts broad geometries using few target anchors\.Dotted curves use only target\-specific ImageNet anchors, while solid curves use posterior anchor models adapted from WordNet with the same target anchors\. Posterior models are strongest at small and moderate target\-anchor budgets, with the advantage narrowing as coverage grows\.

## 6Discussion and Related Work

##### Semantic geometry in contrastive representations\.

Prior work has shown that learned representations often encode semantic structure geometrically, from word\-embedding analogies and latent\-variable accounts of embeddings to linear structure in neural representations\[[Mik\+13](https://arxiv.org/html/2610.06919#bib.bibx17),[Aro\+16](https://arxiv.org/html/2610.06919#bib.bibx4),[SMG19](https://arxiv.org/html/2610.06919#bib.bibx29),[PCV24](https://arxiv.org/html/2610.06919#bib.bibx23),[Par\+25](https://arxiv.org/html/2610.06919#bib.bibx22)\]\. In vision–language models, related work has studied how CLIP\-like representations organize concepts through directions, parts, subspaces, and interpretable coordinates\[[Rad\+21](https://arxiv.org/html/2610.06919#bib.bibx25),[Chu\+25](https://arxiv.org/html/2610.06919#bib.bibx8),[Old\+23](https://arxiv.org/html/2610.06919#bib.bibx21),[Bha\+24](https://arxiv.org/html/2610.06919#bib.bibx7),[GES24](https://arxiv.org/html/2610.06919#bib.bibx12),[Mer\+23](https://arxiv.org/html/2610.06919#bib.bibx16)\]\. Our question is complementary: given that a representation may encode many semantic factors simultaneously, which of them should determine similarity in a particular context? Anchor divergences answer this by representing the semantic context through an anchor distribution, which induces a corresponding Bregman geometry on the fixed representation space\.

##### Image–image comparison in CLIP\-like models\.

CLIP\-style image embeddings are commonly compared using cosine similarity across applications\[[Abb\+25](https://arxiv.org/html/2610.06919#bib.bibx1),[Zhe\+25](https://arxiv.org/html/2610.06919#bib.bibx40),[Den\+25](https://arxiv.org/html/2610.06919#bib.bibx9),[Tiu\+22](https://arxiv.org/html/2610.06919#bib.bibx31)\]\. Several works have proposed alternatives to raw image–image cosine comparison, including cache\-based adaptation, cross\-modal aggregation, text\-based inversion, and methods addressing image–text or intra\-modal mismatch\[[Zha\+22](https://arxiv.org/html/2610.06919#bib.bibx39),[UGA23](https://arxiv.org/html/2610.06919#bib.bibx32),[Yi\+24](https://arxiv.org/html/2610.06919#bib.bibx38),[Mis\+25](https://arxiv.org/html/2610.06919#bib.bibx18),[Bal\+23](https://arxiv.org/html/2610.06919#bib.bibx5),[HW26](https://arxiv.org/html/2610.06919#bib.bibx14)\]\. These works motivate alternatives to the fixed cosine geometry, but do not generally ask how semantic context itself should determine the geometry on a fixed representation space\.

##### Relation to TIP\-X\.

TIP\-X uses text as a bridge for image–image comparison: each image is mapped to a softmax affinity profile over downstream class\-text prompts, and images are compared by the KL divergence between these profiles\[[UGA23](https://arxiv.org/html/2610.06919#bib.bibx32)\]\. This is the same finite\-anchor construction as our softmax anchor divergence when the anchors are the downstream class\-text prompts\. Our contribution begins from this finite construction and studies the geometry determined by an underlying distribution over anchors\. We show that an anchor distribution induces a population Bregman geometry and use this perspective to model and adapt semantic geometry\.

##### Information geometry and metric learning\.

Our construction builds on classical connections between exponential families, KL divergence, and Bregman geometry\[[Efr78](https://arxiv.org/html/2610.06919#bib.bibx10),[Ama82](https://arxiv.org/html/2610.06919#bib.bibx2),[Ban\+05](https://arxiv.org/html/2610.06919#bib.bibx6),[WJ08](https://arxiv.org/html/2610.06919#bib.bibx34),[AC10](https://arxiv.org/html/2610.06919#bib.bibx3)\]\. It is also related to metric learning, which learns task\-specific distances from supervised constraints on the representation space\[[Xin\+02](https://arxiv.org/html/2610.06919#bib.bibx36),[WS09](https://arxiv.org/html/2610.06919#bib.bibx35),[SKP15](https://arxiv.org/html/2610.06919#bib.bibx30),[MA\+17](https://arxiv.org/html/2610.06919#bib.bibx19),[MBL20](https://arxiv.org/html/2610.06919#bib.bibx20),[Rot\+20](https://arxiv.org/html/2610.06919#bib.bibx26)\]\. Anchor divergences instead specify the semantic context through a distribution over semantic text anchors, without requiring labeled image pairs or representation fine\-tuning\. Because this anchor distribution determines the induced geometry, parametric modeling imposes corresponding geometric structure, while Bayesian updating provides a mechanism for semantic adaptation\.

##### Future directions\.

Although our experiments focus on image–image retrieval in vision–language models, the construction is not intrinsically tied to images\. The key ingredient is that a representation in one space defines a softmax distribution over anchors in another space through query–key scores\. The theoretical structure developed here follows from this softmax, or equivalently exponential\-family, form rather than from any image\-specific property\. In this view, a distribution over keys induces a geometry on the query space\. This suggests that anchor divergences extend naturally to other contrastive and multimodal settings with the same structure\. More broadly, this points to the importance of understanding the geometry of softmax\-based models as a general problem in AI: recent work argues that representations defining softmax distributions are naturally studied through information geometry, with implications for probing and steering\[[Par\+26](https://arxiv.org/html/2610.06919#bib.bibx24)\]\. Progress on anchor divergences may therefore be of independent interest beyond retrieval, contributing to a broader theory of semantic geometry in softmax models\.

#### Acknowledgments

This work is supported by ONR grant N00014\-23\-1\-2591\.

## References

- \[Abb\+25\]Reza Abbasi et al\.“Clip under the microscope: A fine\-grained analysis of multi\-object representation”In*2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2025, pp\. 9308–9317IEEE
- \[Ama82\]Shun\-Ichi Amari“Differential geometry of curved exponential families\-curvatures and information loss”In*The Annals of Statistics*JSTOR, 1982, pp\. 357–385
- \[AC10\]Shun\-ichi Amari and Andrzej Cichocki“Information geometry of divergence functions”In*Bulletin of the polish academy of sciences\. Technical sciences*58\.1Polska Akademia Nauk\. Czasopisma i Monografie PAN, 2010, pp\. 183–195
- \[Aro\+16\]Sanjeev Arora et al\.“A latent variable model approach to pmi\-based word embeddings”In*Transactions of the Association for Computational Linguistics*4MIT Press, 2016, pp\. 385–399
- \[Bal\+23\]Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini and Alberto Del“Zero\-shot composed image retrieval with textual inversion”In*2023 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023, pp\. 15292–15301IEEE
- \[Ban\+05\]Arindam Banerjee, Srujana Merugu, Inderjit Dhillon and Joydeep Ghosh“Clustering with Bregman divergences”In*Journal of machine learning research*6\.Oct, 2005, pp\. 1705–1749
- \[Bha\+24\]Usha Bhalla et al\.“Interpreting clip with sparse linear concept embeddings \(splice\)”In*Advances in Neural Information Processing Systems*37, 2024, pp\. 84298–84328
- \[Chu\+25\]Yung\-Sung Chuang et al\.“Meta CLIP 2: A Worldwide Scaling Recipe”In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025URL:[https://openreview\.net/forum?id=aYRNINhNGV](https://openreview.net/forum?id=aYRNINhNGV)
- \[Den\+25\]Stefan Denner et al\.“Leveraging foundation models for content\-based image retrieval in radiology”In*Computers in Biology and Medicine*196Elsevier, 2025, pp\. 110640
- \[Efr78\]Bradley Efron“The geometry of exponential families”In*The Annals of Statistics*JSTOR, 1978, pp\. 362–376
- \[Eng\+25\]Josh Engels et al\.“Not all language model features are one\-dimensionally linear”In*International Conference on Learning Representations*2025, 2025, pp\. 84591–84622
- \[GES24\]Yossi Gandelsman, Alexei Efros and Jacob Steinhardt“Interpreting clip’s image representation via text\-based decomposition”In*International conference on learning representations*2024, 2024, pp\. 18395–18416
- \[Hen\+21\]Dan Hendrycks et al\.“The many faces of robustness: A critical analysis of out\-of\-distribution generalization”In*2021 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2021, pp\. 8320–8329IEEE
- \[HW26\]Jonas Herzog and Yue Wang“Reevaluating the Intra\-Modal Misalignment Hypothesis in CLIP”In*arXiv preprint arXiv:2603\.16100*, 2026
- \[Irv\+19\]Jeremy Irvin et al\.“Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison”In*Proceedings of the AAAI conference on artificial intelligence*33, 2019, pp\. 590–597
- \[Mer\+23\]Jack Merullo, Louis Castricato, Carsten Eickhoff and Ellie Pavlick“Linearly Mapping from Image to Text Space”In*The Eleventh International Conference on Learning Representations*, 2023URL:[https://openreview\.net/forum?id=8tYRqb05pVn](https://openreview.net/forum?id=8tYRqb05pVn)
- \[Mik\+13\]Tom\\’as Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean“Efficient Estimation of Word Representations in Vector Space”In*1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2–4, 2013, Workshop Track Proceedings*, 2013
- \[Mis\+25\]Marco Mistretta et al\.“Cross the gap: Exposing the intra\-modal misalignment in clip via modality inversion”In*International Conference on Learning Representations*2025, 2025, pp\. 77750–77771
- \[MA\+17\]Yair Movshovitz\-Attias et al\.“No fuss distance metric learning using proxies”In*2017 IEEE International Conference on Computer Vision \(ICCV\)*, 2017, pp\. 360–368IEEE
- \[MBL20\]Kevin Musgrave, Serge Belongie and Ser\-Nam Lim“A metric learning reality check”In*European Conference on Computer Vision*, 2020, pp\. 681–699Springer
- \[Old\+23\]James Oldfield et al\.“Parts of speech–grounded subspaces in vision\-language models”In*Advances in Neural Information Processing Systems*36, 2023, pp\. 2700–2724
- \[Par\+25\]Kiho Park, Yo Choe, Yibo Jiang and Victor Veitch“The geometry of categorical and hierarchical concepts in large language models”In*International Conference on Learning Representations*2025, 2025, pp\. 76441–76463
- \[PCV24\]Kiho Park, Yo Choe and Victor Veitch“The Linear Representation Hypothesis and the Geometry of Large Language Models”In*Proceedings of the 41st International Conference on Machine Learning*PMLR, 2024, pp\. 39643–39666URL:[https://proceedings\.mlr\.press/v235/park24c\.html](https://proceedings.mlr.press/v235/park24c.html)
- \[Par\+26\]Kiho Park, Todd Nief, Yo Choe and Victor Veitch“The Information Geometry of Softmax: Probing and Steering”In*Proceedings of the 43rd International Conference on Machine Learning*PMLR, 2026URL:[https://proceedings\.mlr\.press/v306/park26p\.html](https://proceedings.mlr.press/v306/park26p.html)
- \[Rad\+21\]Alec Radford et al\.“Learning transferable visual models from natural language supervision”In*International conference on machine learning*, 2021, pp\. 8748–8763PmLR
- \[Rot\+20\]Karsten Roth et al\.“Revisiting training strategies and generalization performance in deep metric learning”In*International conference on machine learning*, 2020, pp\. 8242–8252PMLR
- \[Rus\+15\]Olga Russakovsky et al\.“Imagenet large scale visual recognition challenge”In*International journal of computer vision*115\.3Springer, 2015, pp\. 211–252
- \[San\+16\]Patsorn Sangkloy, Nathan Burnell, Cusuh Ham and James Hays“The sketchy database: learning to retrieve badly drawn bunnies”In*Acm Transactions on Graphics \(TOG\)*35\.4ACM New York, NY, USA, 2016, pp\. 1–12
- \[SMG19\]Andrew Saxe, James McClelland and Surya Ganguli“A mathematical theory of semantic development in deep neural networks”In*Proceedings of the National Academy of Sciences*116\.23National Academy of Sciences, 2019, pp\. 11537–11546
- \[SKP15\]Florian Schroff, Dmitry Kalenichenko and James Philbin“Facenet: A unified embedding for face recognition and clustering”In*Proceedings of the IEEE conference on computer vision and pattern recognition*, 2015, pp\. 815–823
- \[Tiu\+22\]Ekin Tiu et al\.“Expert\-level detection of pathologies from unannotated chest X\-ray images via self\-supervised learning”In*Nature biomedical engineering*6\.12Nature Publishing Group UK London, 2022, pp\. 1399–1406
- \[UGA23\]Vishaal Udandarao, Ankush Gupta and Samuel Albanie“Sus\-x: Training\-free name\-only transfer of vision\-language models”In*2023 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023, pp\. 2725–2736IEEE
- \[Wah\+11\]Catherine Wah et al\.“The caltech\-ucsd birds\-200\-2011 dataset”, 2011
- \[WJ08\]Martin Wainwright and Michael Jordan“Graphical models, exponential families, and variational inference”In*Foundations and Trends® in Machine Learning*1\.1\-2, 2008, pp\. 1–305
- \[WS09\]Kilian Weinberger and Lawrence Saul“Distance metric learning for large margin nearest neighbor classification\.”In*Journal of machine learning research*10\.2, 2009
- \[Xin\+02\]Eric Xing, Michael Jordan, Stuart Russell and Andrew Ng“Distance Metric Learning with Application to Clustering with Side\-Information”In*Advances in Neural Information Processing Systems*15MIT Press, 2002
- \[Yel\+18\]Sasi Yelamarthi, Shiva Reddy, Ashish Mishra and Anurag Mittal“A zero\-shot framework for sketch based image retrieval”In*European conference on computer vision*, 2018, pp\. 316–333Springer
- \[Yi\+24\]Chao Yi, Lu Ren, De\-Chuan Zhan and Han\-Jia Ye“Leveraging cross\-modal neighbor representation for improved clip classification”In*2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024, pp\. 27392–27401IEEE
- \[Zha\+22\]Renrui Zhang et al\.“Tip\-adapter: Training\-free adaption of clip for few\-shot classification”In*European conference on computer vision*, 2022, pp\. 493–510Springer
- \[Zhe\+25\]Sunyi Zheng et al\.“Benchmarking pathclip for pathology image analysis”In*Journal of Imaging Informatics in Medicine*38\.1Springer, 2025, pp\. 422–438

## Appendix AProof of[Theorem3](https://arxiv.org/html/2610.06919#Thmtheorem3)

See[3](https://arxiv.org/html/2610.06919#Thmtheorem3)

###### Proof\.

We begin by introducing some notation and definitions\. We define

�=��,ℰ:=\{��:�∈𝒮d−1\}⊂Rd\.\\displaystyle\\eta=\\frac\{\\lambda\}\{\\tau\},\\qquad\\mathcal\{E\}:=\\left\\\{\\frac\{\\lambda\}\{\\tau\}:\\lambda\\in\\mathcal\{S\}^\{d\-1\}\\right\\\}\\subset\\mdmathbb R^\{d\}\.Since𝒮d−1\\mathcal\{S\}^\{d\-1\}is compact,ℰ\\mathcal\{E\}is compact as well\. We rewrite the population cumulant generating function as

�\(�\)=log∫e�⊤​q\(d\)=logEq\[e�⊤​\]\.\\displaystyle\\kappa\(\\eta\)=\\log\\int e^\{\\eta^\{\\top\}\\gamma\}\\,q\(d\\gamma\)=\\log\\mdmathbb E\_\{q\}\\left\[e^\{\\eta^\{\\top\}\\gamma\}\\right\]\.For an anchor set�N=\{,1…,\}N\\Gamma\_\{N\}=\\\{\{\}\_\{1\},\\dots,\{\}\_\{N\}\\\}, the cumulant generating function of the empirical measure is

�^N\(�\):=log\(1N∑i=1Ne�⊤i\)\.\\displaystyle\\widehat\{\\kappa\}\_\{N\}\(\\eta\):=\\log\\left\(\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{\\eta^\{\\top\}\{\}\_\{i\}\}\\right\)\.The factor1/N1/Ndoes not affect the induced softmax probabilities, but it is convenient for the law of large numbers\. We also define

ZN\(�\):=1N∑i=1Ne�⊤i,Z\(�\):=Eq\[e�⊤​\],\\displaystyle Z\_\{N\}\(\\eta\):=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{\\eta^\{\\top\}\{\}\_\{i\}\},\\qquad Z\(\\eta\):=\\mdmathbb E\_\{q\}\[e^\{\\eta^\{\\top\}\\gamma\}\],and

MN\(�\):=1N∑i=1Ne�⊤ii,M\(�\):=Eq\[e�⊤​\]\.\\displaystyle M\_\{N\}\(\\eta\):=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\{\}\_\{i\}e^\{\\eta^\{\\top\}\{\}\_\{i\}\},\\qquad M\(\\eta\):=\\mdmathbb E\_\{q\}\[\\gamma e^\{\\eta^\{\\top\}\\gamma\}\]\.Throughout this proof, the anchors are sampled i\.i\.d\. fromqq\.

Now we are ready to prove the theorem\. The first step is to rewrite the discrete softmax family as an empirical exponential family\. Given�N=\{,1…,\}N\\Gamma\_\{N\}=\\\{\{\}\_\{1\},\\dots,\{\}\_\{N\}\\\}and�∈Rd\\eta\\in\\mdmathbb R^\{d\},

pN​\(i\|�\)\\displaystyle p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\)=e�⊤i∑j=1Ne�⊤j\\displaystyle=\\frac\{e^\{\\eta^\{\\top\}\{\}\_\{i\}\}\}\{\\sum\_\{j=1\}^\{N\}e^\{\\eta^\{\\top\}\{\}\_\{j\}\}\}=1Ne�⊤i1N∑j=1Ne�⊤j\\displaystyle=\\frac\{\\frac\{1\}\{N\}e^\{\\eta^\{\\top\}\{\}\_\{i\}\}\}\{\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}e^\{\\eta^\{\\top\}\{\}\_\{j\}\}\}=exp\(�⊤−i�^N\(�\)\)⋅1N\.\\displaystyle=\\exp\\big\(\\eta^\{\\top\}\{\}\_\{i\}\-\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\\big\)\\cdot\\frac\{1\}\{N\}\.Thus, the discrete softmax is exactly the exponential tilt of the empirical anchor measure\.

Let�a=�a/�\\eta\_\{a\}=\\lambda\_\{a\}/\\taufora=1,2a=1,2\. Then

dSM\(�1∥�2\|�N\)\\displaystyle d\_\{\\mathrm\{SM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)=∑i=1NpN​\(i\|�1\)​log⁡pN​\(i\|�1\)pN​\(i\|�2\)\\displaystyle=\\sum\_\{i=1\}^\{N\}p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{1\}\)\\log\\frac\{p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{1\}\)\}\{p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{2\}\)\}=∑i=1NpN\(i\|�1\)\(\(�1−�2\)⊤−i�^N\(�1\)\+�^N\(�2\)\)\.\\displaystyle=\\sum\_\{i=1\}^\{N\}p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{1\}\)\\Big\(\(\\eta\_\{1\}\-\\eta\_\{2\}\)^\{\\top\}\{\}\_\{i\}\-\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{1\}\)\+\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{2\}\)\\Big\)\.Since∑ipN​\(i\|�1\)=1\\sum\_\{i\}p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{1\}\)=1, this becomes

dSM\(�1∥�2\|�N\)\\displaystyle d\_\{\\mathrm\{SM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)=\(�1−�2\)⊤∑i=1NpN\(i\|�1\)−i�^N\(�1\)\+�^N\(�2\)\.\\displaystyle=\(\\eta\_\{1\}\-\\eta\_\{2\}\)^\{\\top\}\\sum\_\{i=1\}^\{N\}p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\_\{1\}\)\{\}\_\{i\}\-\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{1\}\)\+\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{2\}\)\.\(A\.1\)But

∇�^N\(�\)=∑i=1Nie�⊤i∑j=1Ne�⊤j=∑i=1NpN\(i\|�\),i\\displaystyle\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\)=\\frac\{\\sum\_\{i=1\}^\{N\}\{\}\_\{i\}e^\{\\eta^\{\\top\}\{\}\_\{i\}\}\}\{\\sum\_\{j=1\}^\{N\}e^\{\\eta^\{\\top\}\{\}\_\{j\}\}\}=\\sum\_\{i=1\}^\{N\}p\_\{N\}\(i\\penalty\\ \|\\penalty\\ \\eta\)\{\}\_\{i\},so[EquationA\.1](https://arxiv.org/html/2610.06919#A1.E1)can be written as

dSM\(�1∥�2\|�N\)\\displaystyle d\_\{\\mathrm\{SM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)=�^N\(�2\)−�^N\(�1\)\+\(�1−�2\)⊤∇�^N\(�1\)\.\\displaystyle=\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{2\}\)\-\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{1\}\)\+\(\\eta\_\{1\}\-\\eta\_\{2\}\)^\{\\top\}\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{1\}\)\.\(A\.2\)Equivalently,

dSM\(�1∥�2\|�N\)=B�^N\(�2∥�1\),\\displaystyle d\_\{\\mathrm\{SM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)=B\_\{\\widehat\{\\kappa\}\_\{N\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\),\(A\.3\)whereBf\(y∥x\):=f\(y\)−f\(x\)−∇f\(x\)⊤\(y−x\)B\_\{f\}\(y\\\|x\):=f\(y\)\-f\(x\)\-\\nabla f\(x\)^\{\\top\}\(y\-x\)\.

We now show uniform convergence ofZNZ\_\{N\}toZZonℰ\\mathcal\{E\}\. Since‖�‖=1/�\\\|\\eta\\\|=1/\\taufor�∈ℰ\\eta\\in\\mathcal\{E\}and‖‖=1\\\|\\gamma\\\|=1, we have

0<e�⊤​≤e1/�for all�∈ℰ,∈𝒮d−1\.\\displaystyle 0<e^\{\\eta^\{\\top\}\\gamma\}\\leq e^\{1/\\tau\}\\qquad\\text\{for all \}\\eta\\in\\mathcal\{E\},\\ \\gamma\\in\\mathcal\{S\}^\{d\-1\}\.Moreover, for any�,�′∈ℰ\\eta,\\eta^\{\\prime\}\\in\\mathcal\{E\}and∈𝒮d−1\\gamma\\in\\mathcal\{S\}^\{d\-1\},

\|e�⊤​−e\(�′\)⊤​\|\\displaystyle\\left\|e^\{\\eta^\{\\top\}\\gamma\}\-e^\{\(\\eta^\{\\prime\}\)^\{\\top\}\\gamma\}\\right\|≤e1/�​\|\(�−�′\)⊤​\|≤e1/�​‖�−�′‖\.\\displaystyle\\leq e^\{1/\\tau\}\\,\|\(\\eta\-\\eta^\{\\prime\}\)^\{\\top\}\\gamma\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|\.Therefore, bothZNZ\_\{N\}andZZare Lipschitz onℰ\\mathcal\{E\}with deterministic Lipschitz constant at moste1/�e^\{1/\\tau\}:

\|ZN​\(�\)−ZN​\(�′\)\|≤e1/�​‖�−�′‖,\|Z⁡\(�\)−Z⁡\(�′\)\|≤e1/�​‖�−�′‖\.\\displaystyle\|Z\_\{N\}\(\\eta\)\-Z\_\{N\}\(\\eta^\{\\prime\}\)\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|,\\qquad\|Z\(\\eta\)\-Z\(\\eta^\{\\prime\}\)\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|\.
Fix"\>0\\varepsilon\>0, and letT"⊂ℰT\_\{\\varepsilon\}\\subset\\mathcal\{E\}be a finite"\\varepsilon\-net of the compact setℰ\\mathcal\{E\}\. For each fixed�∈T"\\eta\\in T\_\{\\varepsilon\}, the strong law of large numbers gives

ZN​\(�\)→a\.s\.Z​\(�\)\.\\displaystyle Z\_\{N\}\(\\eta\)\\xrightarrow\{\\mathrm\{a\.s\.\}\}Z\(\\eta\)\.SinceT"T\_\{\\varepsilon\}is finite, it follows that

max�∈T"⁡\|ZN​\(�\)−Z⁡\(�\)\|→a\.s\.0\.\\displaystyle\\max\_\{\\eta\\in T\_\{\\varepsilon\}\}\|Z\_\{N\}\(\\eta\)\-Z\(\\eta\)\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.Now let�∈ℰ\\eta\\in\\mathcal\{E\}, and choose�"∈T"\\eta\_\{\\varepsilon\}\\in T\_\{\\varepsilon\}with‖�−�"‖≤"\\\|\\eta\-\\eta\_\{\\varepsilon\}\\\|\\leq\\varepsilon\. Then

\|ZN​\(�\)−Z​\(�\)\|\\displaystyle\|Z\_\{N\}\(\\eta\)\-Z\(\\eta\)\|≤\|ZN​\(�\)−ZN​\(�"\)\|\+\|ZN​\(�"\)−Z⁡\(�"\)\|\+\|Z⁡\(�"\)−Z⁡\(�\)\|\\displaystyle\\leq\|Z\_\{N\}\(\\eta\)\-Z\_\{N\}\(\\eta\_\{\\varepsilon\}\)\|\+\|Z\_\{N\}\(\\eta\_\{\\varepsilon\}\)\-Z\(\\eta\_\{\\varepsilon\}\)\|\+\|Z\(\\eta\_\{\\varepsilon\}\)\-Z\(\\eta\)\|≤2​e1/�​"\+max�′∈T"⁡\|ZN​\(�′\)−Z⁡\(�′\)\|\.\\displaystyle\\leq 2e^\{1/\\tau\}\\varepsilon\+\\max\_\{\\eta^\{\\prime\}\\in T\_\{\\varepsilon\}\}\|Z\_\{N\}\(\\eta^\{\\prime\}\)\-Z\(\\eta^\{\\prime\}\)\|\.Taking the supremum over�∈ℰ\\eta\\in\\mathcal\{E\}yields

sup�∈ℰ\|ZN​\(�\)−Z⁡\(�\)\|≤2​e1/�​"\+max�′∈T"⁡\|ZN​\(�′\)−Z⁡\(�′\)\|\.\\displaystyle\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|Z\_\{N\}\(\\eta\)\-Z\(\\eta\)\|\\leq 2e^\{1/\\tau\}\\varepsilon\+\\max\_\{\\eta^\{\\prime\}\\in T\_\{\\varepsilon\}\}\|Z\_\{N\}\(\\eta^\{\\prime\}\)\-Z\(\\eta^\{\\prime\}\)\|\.Hence, almost surely,

lim supN→∞sup�∈ℰ\|ZN​\(�\)−Z⁡\(�\)\|≤2​e1/�​"\.\\displaystyle\\limsup\_\{N\\to\\infty\}\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|Z\_\{N\}\(\\eta\)\-Z\(\\eta\)\|\\leq 2e^\{1/\\tau\}\\varepsilon\.Since"\>0\\varepsilon\>0is arbitrary, we conclude

sup�∈ℰ\|ZN​\(�\)−Z⁡\(�\)\|→a\.s\.0\.\\displaystyle\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|Z\_\{N\}\(\\eta\)\-Z\(\\eta\)\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.\(A\.4\)
Using the same Lipschitz argument, for any�,�′∈ℰ\\eta,\\eta^\{\\prime\}\\in\\mathcal\{E\}and∈𝒮d−1\\gamma\\in\\mathcal\{S\}^\{d\-1\},

∥e�⊤​−e\(�′\)⊤​∥≤e1/�∥�−�′∥\.\\displaystyle\\\|\\gamma e^\{\\eta^\{\\top\}\\gamma\}\-\\gamma e^\{\(\\eta^\{\\prime\}\)^\{\\top\}\\gamma\}\\\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|\.Therefore, bothMNM\_\{N\}andMMare Lipschitz onℰ\\mathcal\{E\}with deterministic Lipschitz constant at moste1/�e^\{1/\\tau\}:

‖MN​\(�\)−MN​\(�′\)‖≤e1/�​‖�−�′‖,‖M⁡\(�\)−M⁡\(�′\)‖≤e1/�​‖�−�′‖\.\\displaystyle\\\|M\_\{N\}\(\\eta\)\-M\_\{N\}\(\\eta^\{\\prime\}\)\\\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|,\\qquad\\\|M\(\\eta\)\-M\(\\eta^\{\\prime\}\)\\\|\\leq e^\{1/\\tau\}\\\|\\eta\-\\eta^\{\\prime\}\\\|\.Applying the same finite\-net argument componentwise gives

sup�∈ℰ‖MN​\(�\)−M⁡\(�\)‖→a\.s\.0\.\\displaystyle\\sup\_\{\\eta\\in\\mathcal\{E\}\}\\\|M\_\{N\}\(\\eta\)\-M\(\\eta\)\\\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.\(A\.5\)
Since

∥e�⊤​∥≤e1/�for all�∈ℰ,∈𝒮d−1,\\displaystyle\\\|\\gamma e^\{\\eta^\{\\top\}\\gamma\}\\\|\\leq e^\{1/\\tau\}\\qquad\\text\{for all \}\\eta\\in\\mathcal\{E\},\\ \\gamma\\in\\mathcal\{S\}^\{d\-1\},dominated convergence justifies differentiation under the expectation, so

∇Z\(�\)=Eq\[e�⊤​\]=M\(�\)\.\\displaystyle\\nabla Z\(\\eta\)=\\mdmathbb E\_\{q\}\[\\gamma e^\{\\eta^\{\\top\}\\gamma\}\]=M\(\\eta\)\.Hence,

∇�​\(�\)=∇Z​\(�\)Z⁡\(�\)=M⁡\(�\)Z⁡\(�\),∇�^N​\(�\)=MN​\(�\)ZN​\(�\)\.\\displaystyle\\nabla\\kappa\(\\eta\)=\\frac\{\\nabla Z\(\\eta\)\}\{Z\(\\eta\)\}=\\frac\{M\(\\eta\)\}\{Z\(\\eta\)\},\\qquad\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\)=\\frac\{M\_\{N\}\(\\eta\)\}\{Z\_\{N\}\(\\eta\)\}\.
By[eq\.A\.4](https://arxiv.org/html/2610.06919#A1.E4),ZN→ZZ\_\{N\}\\to Zuniformly almost surely onℰ\\mathcal\{E\}\. Also,Z⁡\(�\)\>0Z\(\\eta\)\>0for every�\\eta, andZZis continuous on compactℰ\\mathcal\{E\}, so

m:=inf�∈ℰZ⁡\(�\)\>0\.\\displaystyle m:=\\inf\_\{\\eta\\in\\mathcal\{E\}\}Z\(\\eta\)\>0\.Therefore, almost surely for all sufficiently largeNN,

inf�∈ℰZN​\(�\)≥m2\>0\.\\displaystyle\\inf\_\{\\eta\\in\\mathcal\{E\}\}Z\_\{N\}\(\\eta\)\\geq\\frac\{m\}\{2\}\>0\.Becauselog\\logis Lipschitz on\[m/2,∞\)\[m/2,\\infty\),[EquationA\.4](https://arxiv.org/html/2610.06919#A1.E4)implies

sup�∈ℰ\|�^N​\(�\)−�​\(�\)\|→a\.s\.0\.\\displaystyle\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\-\\kappa\(\\eta\)\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.\(A\.6\)
Using[EquationA\.4](https://arxiv.org/html/2610.06919#A1.E4)and[EquationA\.5](https://arxiv.org/html/2610.06919#A1.E5), together with the lower bound on the denominators, we obtain

sup�∈ℰ‖∇�^N​\(�\)−∇�​\(�\)‖→a\.s\.0\.\\displaystyle\\sup\_\{\\eta\\in\\mathcal\{E\}\}\\\|\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\-\\nabla\\kappa\(\\eta\)\\\|\\xrightarrow\{\\mathrm\{a\.s\.\}\}0\.\(A\.7\)
Recall that

B�\(�2∥�1\)=�\(�2\)−�\(�1\)−∇�\(�1\)⊤\(�2−�1\)=�\(�2\)−�\(�1\)\+\(�1−�2\)⊤∇�\(�1\)\.\\displaystyle B\_\{\\kappa\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)=\\kappa\(\\eta\_\{2\}\)\-\\kappa\(\\eta\_\{1\}\)\-\\nabla\\kappa\(\\eta\_\{1\}\)^\{\\top\}\(\\eta\_\{2\}\-\\eta\_\{1\}\)=\\kappa\(\\eta\_\{2\}\)\-\\kappa\(\\eta\_\{1\}\)\+\(\\eta\_\{1\}\-\\eta\_\{2\}\)^\{\\top\}\\nabla\\kappa\(\\eta\_\{1\}\)\.Hence,

sup�1,�2∈ℰ\|dSM\(��1∥��2\|�N\)−B�\(�2∥�1\)\|\\displaystyle\\sup\_\{\\eta\_\{1\},\\eta\_\{2\}\\in\\mathcal\{E\}\}\\left\|d\_\{\\mathrm\{SM\}\}\(\\tau\\eta\_\{1\}\\\|\\tau\\eta\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)\-B\_\{\\kappa\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\\right\|≤2​sup�∈ℰ\|�^N​\(�\)−�​\(�\)\|\+sup�1,�2∈ℰ\|\(�1−�2\)⊤​\(∇�^N​\(�1\)−∇�​\(�1\)\)\|\.\\displaystyle\\leq 2\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\-\\kappa\(\\eta\)\|\+\\sup\_\{\\eta\_\{1\},\\eta\_\{2\}\\in\\mathcal\{E\}\}\\left\|\(\\eta\_\{1\}\-\\eta\_\{2\}\)^\{\\top\}\\big\(\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\_\{1\}\)\-\\nabla\\kappa\(\\eta\_\{1\}\)\\big\)\\right\|\.Sinceℰ=�−1​𝒮d−1\\mathcal\{E\}=\\tau^\{\-1\}\\mathcal\{S\}^\{d\-1\},

sup�1,�2∈ℰ‖�1−�2‖≤2�\.\\displaystyle\\sup\_\{\\eta\_\{1\},\\eta\_\{2\}\\in\\mathcal\{E\}\}\\\|\\eta\_\{1\}\-\\eta\_\{2\}\\\|\\leq\\frac\{2\}\{\\tau\}\.Therefore,

sup�1,�2∈ℰ\|dSM\(��1∥��2\|�N\)−B�\(�2∥�1\)\|\\displaystyle\\sup\_\{\\eta\_\{1\},\\eta\_\{2\}\\in\\mathcal\{E\}\}\\left\|d\_\{\\mathrm\{SM\}\}\(\\tau\\eta\_\{1\}\\\|\\tau\\eta\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)\-B\_\{\\kappa\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\\right\|≤2​sup�∈ℰ\|�^N​\(�\)−�​\(�\)\|\+2�​sup�∈ℰ‖∇�^N​\(�\)−∇�​\(�\)‖\.\\displaystyle\\quad\\quad\\quad\\quad\\leq 2\\sup\_\{\\eta\\in\\mathcal\{E\}\}\|\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\-\\kappa\(\\eta\)\|\+\\frac\{2\}\{\\tau\}\\sup\_\{\\eta\\in\\mathcal\{E\}\}\\\|\\nabla\\widehat\{\\kappa\}\_\{N\}\(\\eta\)\-\\nabla\\kappa\(\\eta\)\\\|\.By[EquationA\.6](https://arxiv.org/html/2610.06919#A1.E6)and[EquationA\.7](https://arxiv.org/html/2610.06919#A1.E7), the right\-hand side converges almost surely to zero\. Replacing�a=�a/�\\eta\_\{a\}=\\lambda\_\{a\}/\\taugives

sup�1,�2∈𝒮d−1\|dSM\(�1∥�2\|�N\)−B�\(�2�∥�1�\)\|→N→∞a\.s\.0\.\\displaystyle\\sup\_\{\\lambda\_\{1\},\\lambda\_\{2\}\\in\\mathcal\{S\}^\{d\-1\}\}\\left\|d\_\{\\mathrm\{SM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\\penalty\\ \|\\penalty\\ \\Gamma\_\{N\}\)\-B\_\{\\kappa\}\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\)\\right\|\\xrightarrow\[N\\to\\infty\]\{\\mathrm\{a\.s\.\}\}0\.
This proves the claim\. ∎

## Appendix BStructured approximations to the population anchor geometry

The population anchor divergence is determined by the anchor distribution through its cumulant generating function\. To obtain tractable estimators of this geometry, we consider structured parametric approximations to the anchor distribution\. These models should be understood as pragmatic structured surrogates for the population anchor geometry, not as literal distributional models of normalized contrastive embeddings\. The Gaussian and Gaussian\-mixture models place mass off the unit sphere, while the latent low\-rank models additionally idealize the semantically relevant variation as concentrated in a lower\-dimensional subspace\. We use these models because they yield tractable geometries with useful statistical tradeoffs in fidelity, robustness, and sample efficiency\.

### B\.1General framework

We begin with a latent semantic variablez∈𝒵⊆Rkz\\in\\mathcal\{Z\}\\subseteq\\mdmathbb R^\{k\}, representing text\-side semantic information, and a corresponding latent natural parameter�∈ℋ⊆Rk\\eta\\in\\mathcal\{H\}\\subseteq\\mdmathbb R^\{k\}, representing the image\-side semantic parameter\. Given a base measureqZq\_\{Z\}on𝒵\\mathcal\{Z\}, we let�\\etaparameterizezzthrough the exponential family

p⁡\(d​z\|�\)=exp⁡\(�⊤​z−�Z​\(�\)\)​qZ​\(d​z\),\\displaystyle p\(dz\\penalty\\ \|\\penalty\\ \\eta\)=\\exp\\\!\\left\(\\eta^\{\\top\}z\-\\kappa\_\{Z\}\(\\eta\)\\right\)q\_\{Z\}\(dz\),\(B\.1\)whose cumulant generating function is

�Z\(�\):=log∫exp\(�⊤z\)qZ\(dz\)\.\\displaystyle\\kappa\_\{Z\}\(\\eta\):=\\log\\int\\exp\(\\eta^\{\\top\}z\)\\,q\_\{Z\}\(dz\)\.\(B\.2\)The associated latent semantic geometry is the Bregman divergence

dZ\(�1∥�2\):=B�Z\(�2∥�1\)\.\\displaystyle d\_\{Z\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\):=B\_\{\\kappa\_\{Z\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\.\(B\.3\)
To relate this latent construction to the anchor divergence used in the main text, let

=T⁡\(z\),\\displaystyle\\gamma=T\(z\),\(B\.4\)whereTTmaps latent text\-side semantics into ambient text embeddings\.�\\lambdais then defined as the reparameterization of�\\etaso that�/�\\lambda/\\tauis the natural parameter for the exponential family with sufficient statistic\\gamma, aligning with the CLIP scoring rule\. We consider two cases:

##### Identity case\.

WhenT=IdT=\\mathrm\{Id\}, we havez=z=\\gamma, so

p⁡\(d​\|�\)=exp⁡\(�⊤​−�Z​\(�\)\)​qZ​\(d​\),\\displaystyle p\(d\\gamma\\penalty\\ \|\\penalty\\ \\eta\)=\\exp\\\!\\left\(\\eta^\{\\top\}\\gamma\-\\kappa\_\{Z\}\(\\eta\)\\right\)q\_\{Z\}\(d\\gamma\),\(B\.5\)and hence��=�\\frac\{\\lambda\}\{\\tau\}=\\eta\. In this case,[EquationB\.2](https://arxiv.org/html/2610.06919#A2.E2)reduces to the cumulant generating function of the anchor distribution, and[EquationB\.3](https://arxiv.org/html/2610.06919#A2.E3)recovers the population anchor divergence from the main text\.

##### Low\-rank affine case\.

For the latent low\-rank models, we take

T⁡\(z\)=�\+U​z,\\displaystyle T\(z\)=\\mu\+Uz,\(B\.6\)whereU∈Rd×kU\\in\\mdmathbb R^\{d\\times k\}has orthonormal columns\. Thus, latent semantics are realized in the affine subspace�\+Im⁡\(U\)⊂Rd\\mu\+\\operatorname\{Im\}\(U\)\\subset\\mdmathbb R^\{d\}\. SinceTTis injective onto this subspace, the latent exponential family induces a pushed\-forward family on ambient anchors\. Writingq=T\#​qZq=T\_\{\\\#\}q\_\{Z\}for the pushforward base measure, we obtain

p⁡\(d​\|�\)=exp⁡\(�⊤​U⊤​\(−�\)−�Z​\(�\)\)​q​\(d​\)\.\\displaystyle p\(d\\gamma\\penalty\\ \|\\penalty\\ \\eta\)=\\exp\\\!\\left\(\\eta^\{\\top\}U^\{\\top\}\(\\gamma\-\\mu\)\-\\kappa\_\{Z\}\(\\eta\)\\right\)q\(d\\gamma\)\.\(B\.7\)To align this family with the CLIP score, we parameterize the image side by

��=U​�\.\\displaystyle\\frac\{\\lambda\}\{\\tau\}=U\\eta\.\(B\.8\)So the pushed\-forward family can be written in the score\-aligned form

p⁡\(d​\|�\)=exp⁡\(1�​�⊤​−\[�Z​\(1�​U⊤​�\)\+1�​�⊤​�\]\)​q​\(d​\)\.\\displaystyle p\(d\\gamma\\penalty\\ \|\\penalty\\ \\lambda\)=\\exp\\\!\\left\(\\frac\{1\}\{\\tau\}\\lambda^\{\\top\}\\gamma\-\\Bigl\[\\kappa\_\{Z\}\\\!\\left\(\\frac\{1\}\{\\tau\}U^\{\\top\}\\lambda\\right\)\+\\frac\{1\}\{\\tau\}\\lambda^\{\\top\}\\mu\\Bigr\]\\right\)q\(d\\gamma\)\.\(B\.9\)
The relation�/�=U​�\\lambda/\\tau=U\\etadefines an exact inverse only on the semantic subspaceIm⁡\(U\)\\operatorname\{Im\}\(U\)\. Because the low\-rank models are idealized approximations, general ambient image embeddings need not lie exactly in this subspace, so we use a model\-specific extended inverse or projection map�​\(�\)\\eta\(\\lambda\)\. The induced image–image semantic divergence is then defined by pulling back the latent geometry:

d\(�1∥�2\):=dZ\(�\(�1\)∥�\(�2\)\)=B�Z\(�\(�2\)∥�\(�1\)\)\.\\displaystyle d\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\):=d\_\{Z\}\\bigl\(\\eta\(\\lambda\_\{1\}\)\\\|\\eta\(\\lambda\_\{2\}\)\\bigr\)=B\_\{\\kappa\_\{Z\}\}\\bigl\(\\eta\(\\lambda\_\{2\}\)\\\|\\eta\(\\lambda\_\{1\}\)\\bigr\)\.\(B\.10\)Thus the semantic geometry is computed in latent space, while ambient text and image embeddings provide its realization and parameterization\.

### B\.2Gaussian anchor model

We first recover the ambient\-space Gaussian model as the identity case of the general framework\. Take

z∈Rd,qZ=𝒩⁡\(�,�\),T=Id,\\displaystyle z\\in\\mdmathbb R^\{d\},\\qquad q\_\{Z\}=\\mathcal\{N\}\(\\mu,\\Sigma\),\\qquad T=\\mathrm\{Id\},\(B\.11\)so thatz=z=\\gamma\. By the identity\-case discussion in[SectionB\.1](https://arxiv.org/html/2610.06919#A2.SS1), the image\-side natural parameter is

�=��\.\\displaystyle\\eta=\\frac\{\\lambda\}\{\\tau\}\.\(B\.12\)Thus the Gaussian anchor model is simply the case in which the latent semantic space coincides with the ambient anchor space\.

###### Proposition 6\(Gaussian anchor model\)\.

For the Gaussian anchor model,

dG\(�1∥�2\)=12​�2\(�1−�2\)⊤�\(�1−�2\)\.d\_\{\\mathrm\{G\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}\\Sigma\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)\.\(B\.13\)

###### Proof\.

ForqZ=𝒩⁡\(�,�\)q\_\{Z\}=\\mathcal\{N\}\(\\mu,\\Sigma\), the cumulant generating function is

�Z​\(�\)\\displaystyle\\kappa\_\{Z\}\(\\eta\)=log∫exp\(�⊤z\)𝒩\(dz;�,�\)\\displaystyle=\\log\\int\\exp\(\\eta^\{\\top\}z\)\\,\\mathcal\{N\}\(dz;\\mu,\\Sigma\)\(B\.14\)=�⊤​�\+12​�⊤​�​�\.\\displaystyle=\\mu^\{\\top\}\\eta\+\\frac\{1\}\{2\}\\eta^\{\\top\}\\Sigma\\eta\.\(B\.15\)Hence

∇�Z​\(�\)=�\+�​�\.\\displaystyle\\nabla\\kappa\_\{Z\}\(\\eta\)=\\mu\+\\Sigma\\eta\.\(B\.16\)Substituting into[EquationB\.3](https://arxiv.org/html/2610.06919#A2.E3)gives

dZ\(�1∥�2\)\\displaystyle d\_\{Z\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\)=B�Z\(�2∥�1\)\\displaystyle=B\_\{\\kappa\_\{Z\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\(B\.17\)=12​\(�2−�1\)⊤​�​\(�2−�1\)\.\\displaystyle=\\frac\{1\}\{2\}\(\\eta\_\{2\}\-\\eta\_\{1\}\)^\{\\top\}\\Sigma\(\\eta\_\{2\}\-\\eta\_\{1\}\)\.\(B\.18\)Using�​\(�\)=�/�\\eta\(\\lambda\)=\\lambda/\\tauin[EquationB\.10](https://arxiv.org/html/2610.06919#A2.E10),

dG\(�1∥�2\)\\displaystyle d\_\{\\mathrm\{G\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=dZ\(�1�∥�2�\)\\displaystyle=d\_\{Z\}\\\!\\left\(\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\right\)\(B\.19\)=12​�2​\(�1−�2\)⊤​�​\(�1−�2\),\\displaystyle=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}\\Sigma\(\\lambda\_\{1\}\-\\lambda\_\{2\}\),\(B\.20\)which proves[EquationB\.13](https://arxiv.org/html/2610.06919#A2.E13)\. ∎

### B\.3Latent Gaussian model

We now impose Gaussian structure only on a lower\-dimensional latent semantic variable\. Take

z∈Rk,qZ=𝒩⁡\(0,Ik\),T⁡\(z\)=�\+U​z,\\displaystyle z\\in\\mdmathbb R^\{k\},\\qquad q\_\{Z\}=\\mathcal\{N\}\(0,I\_\{k\}\),\\qquad T\(z\)=\\mu\+Uz,\(B\.21\)whereU∈Rd×kU\\in\\mdmathbb R^\{d\\times k\}has orthonormal columns\. By the low\-rank affine construction in[SectionB\.1](https://arxiv.org/html/2610.06919#A2.SS1), the image\-side parameter satisfies

��=U​�,\\displaystyle\\frac\{\\lambda\}\{\\tau\}=U\\eta,\(B\.22\)and for a general ambient image embedding we use the canonical projection

�​\(�\)=1�​U⊤​�\.\\displaystyle\\eta\(\\lambda\)=\\frac\{1\}\{\\tau\}U^\{\\top\}\\lambda\.\(B\.23\)Thus the image–image divergence is obtained by pulling back the latent Gaussian geometry to the semantic subspace\.

###### Proposition 7\(Latent Gaussian model\)\.

For the latent Gaussian model with semantic subspaceUU,

dLG\(�1∥�2\)=12​�2\(�1−�2\)⊤UU⊤\(�1−�2\)\.d\_\{\\mathrm\{LG\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}UU^\{\\top\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)\.\(B\.24\)

###### Proof\.

For the standard Gaussian base measure onRk\\mdmathbb R^\{k\},

�Z​\(�\)\\displaystyle\\kappa\_\{Z\}\(\\eta\)=log∫exp\(�⊤z\)𝒩\(dz;0,Ik\)\\displaystyle=\\log\\int\\exp\(\\eta^\{\\top\}z\)\\,\\mathcal\{N\}\(dz;0,I\_\{k\}\)\(B\.25\)=12​�⊤​�\.\\displaystyle=\\frac\{1\}\{2\}\\eta^\{\\top\}\\eta\.\(B\.26\)Hence

∇�Z​\(�\)=�,\\displaystyle\\nabla\\kappa\_\{Z\}\(\\eta\)=\\eta,\(B\.27\)and therefore

dZ\(�1∥�2\)=B�Z\(�2∥�1\)=12∥�2−�1∥22\.\\displaystyle d\_\{Z\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\)=B\_\{\\kappa\_\{Z\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)=\\frac\{1\}\{2\}\\\|\\eta\_\{2\}\-\\eta\_\{1\}\\\|\_\{2\}^\{2\}\.\(B\.28\)Using[EquationB\.10](https://arxiv.org/html/2610.06919#A2.E10)together with[EquationB\.23](https://arxiv.org/html/2610.06919#A2.E23),

dLG\(�1∥�2\)\\displaystyle d\_\{\\mathrm\{LG\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=12​‖1�​U⊤​�2−1�​U⊤​�1‖22\\displaystyle=\\frac\{1\}\{2\}\\left\\\|\\frac\{1\}\{\\tau\}U^\{\\top\}\\lambda\_\{2\}\-\\frac\{1\}\{\\tau\}U^\{\\top\}\\lambda\_\{1\}\\right\\\|\_\{2\}^\{2\}\(B\.29\)=12​�2​‖U⊤​\(�2−�1\)‖22\\displaystyle=\\frac\{1\}\{2\\tau^\{2\}\}\\\|U^\{\\top\}\(\\lambda\_\{2\}\-\\lambda\_\{1\}\)\\\|\_\{2\}^\{2\}\(B\.30\)=12​�2​\(�1−�2\)⊤​U​U⊤​\(�1−�2\),\\displaystyle=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}UU^\{\\top\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\),\(B\.31\)which proves[EquationB\.24](https://arxiv.org/html/2610.06919#A2.E24)\. ∎

### B\.4Latent spherical model

The latent spherical model keeps only directional semantic variation within the same low\-dimensional semantic subspace\. We take

z∈Sk−1,�∈1�​Sk−1,qZ=Unif⁡\(Sk−1\),T⁡\(z\)=�\+U​z,\\displaystyle z\\in\\mdmathbb S^\{k\-1\},\\qquad\\eta\\in\\frac\{1\}\{\\tau\}\\mdmathbb S^\{k\-1\},\\qquad q\_\{Z\}=\\mathrm\{Unif\}\(\\mdmathbb S^\{k\-1\}\),\\qquad T\(z\)=\\mu\+Uz,\(B\.32\)whereU∈Rd×kU\\in\\mdmathbb R^\{d\\times k\}has orthonormal columns\. On the semantic subspace, the low\-rank affine construction in[SectionB\.1](https://arxiv.org/html/2610.06919#A2.SS1)gives

��=U​�\.\\displaystyle\\frac\{\\lambda\}\{\\tau\}=U\\eta\.\(B\.33\)For a general ambient image embedding, we extend this parameterization by retaining the direction of its projection while fixing the latent radius to1/�1/\\tau:

�​\(�\):=1�​U⊤​�‖U⊤​�‖2,whenever​U⊤​�≠0\.\\displaystyle\\eta\(\\lambda\):=\\frac\{1\}\{\\tau\}\\frac\{U^\{\\top\}\\lambda\}\{\\\|U^\{\\top\}\\lambda\\\|\_\{2\}\},\\qquad\\text\{whenever \}U^\{\\top\}\\lambda\\neq 0\.\(B\.34\)Thus the image–image divergence is obtained by pulling back the latent directional geometry to the semantic subspace\.

###### Proposition 8\(Latent spherical model\)\.

For the latent spherical model with semantic subspaceUU, and assumingU⊤​�1≠0U^\{\\top\}\\lambda\_\{1\}\\neq 0andU⊤​�2≠0U^\{\\top\}\\lambda\_\{2\}\\neq 0,

dLS\(�1∥�2\)=1�Ak\(1�\)\(1−\(U⊤​�1\)⊤​\(U⊤​�2\)‖U⊤​�1‖2​‖U⊤​�2‖2\),d\_\{\\mathrm\{LS\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{\\tau\}A\_\{k\}\\left\(\\frac\{1\}\{\\tau\}\\right\)\\left\(1\-\\frac\{\(U^\{\\top\}\\lambda\_\{1\}\)^\{\\top\}\(U^\{\\top\}\\lambda\_\{2\}\)\}\{\\\|U^\{\\top\}\\lambda\_\{1\}\\\|\_\{2\}\\,\\\|U^\{\\top\}\\lambda\_\{2\}\\\|\_\{2\}\}\\right\),\(B\.35\)where

Ak​\(r\)=Ik/2​\(r\)Ik/2−1​\(r\),\\displaystyle A\_\{k\}\(r\)=\\frac\{I\_\{k/2\}\(r\)\}\{I\_\{k/2\-1\}\(r\)\},andI�I\_\{\\nu\}denotes the modified Bessel function of the first kind\.

###### Proof\.

For the uniform base measure onSk−1\\mdmathbb S^\{k\-1\}, the cumulant generating function depends only on the norm:

�Z\(�\)=\(∥�∥2\)k,\\displaystyle\\kappa\_\{Z\}\(\\eta\)=\{\}\_\{k\}\(\\\|\\eta\\\|\_\{2\}\),\(B\.36\)wherekis the usual von Mises–Fisher log\-normalizer\. Its gradient is radial:

∇�Z​\(�\)=Ak​\(‖�‖2\)​�‖�‖2\.\\displaystyle\\nabla\\kappa\_\{Z\}\(\\eta\)=A\_\{k\}\(\\\|\\eta\\\|\_\{2\}\)\\frac\{\\eta\}\{\\\|\\eta\\\|\_\{2\}\}\.\(B\.37\)Now let�1,�2∈1�​Sk−1\\eta\_\{1\},\\eta\_\{2\}\\in\\frac\{1\}\{\\tau\}\\mdmathbb S^\{k\-1\}and writer=1/�r=1/\\tau\. Since‖�1‖2=‖�2‖2=r\\\|\\eta\_\{1\}\\\|\_\{2\}=\\\|\\eta\_\{2\}\\\|\_\{2\}=r,

�Z\(�1\)=�Z\(�2\)=\(r\)k,∇�Z\(�1\)=Ak\(r\)�1r\.\\displaystyle\\kappa\_\{Z\}\(\\eta\_\{1\}\)=\\kappa\_\{Z\}\(\\eta\_\{2\}\)=\{\}\_\{k\}\(r\),\\qquad\\nabla\\kappa\_\{Z\}\(\\eta\_\{1\}\)=A\_\{k\}\(r\)\\frac\{\\eta\_\{1\}\}\{r\}\.\(B\.38\)Therefore,

dZ\(�1∥�2\)\\displaystyle d\_\{Z\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\)=B�Z\(�2∥�1\)\\displaystyle=B\_\{\\kappa\_\{Z\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\(B\.39\)=�Z\(�2\)−�Z\(�1\)−∇�Z\(�1\)⊤\(�2−�1\)\\displaystyle=\\kappa\_\{Z\}\(\\eta\_\{2\}\)\-\\kappa\_\{Z\}\(\\eta\_\{1\}\)\-\\nabla\\kappa\_\{Z\}\(\\eta\_\{1\}\)^\{\\top\}\(\\eta\_\{2\}\-\\eta\_\{1\}\)\(B\.40\)=−Ak​\(r\)r​�1⊤​\(�2−�1\)\\displaystyle=\-\\frac\{A\_\{k\}\(r\)\}\{r\}\\eta\_\{1\}^\{\\top\}\(\\eta\_\{2\}\-\\eta\_\{1\}\)\(B\.41\)=r​Ak​\(r\)​\(1−�1⊤​�2r2\)\.\\displaystyle=rA\_\{k\}\(r\)\\left\(1\-\\frac\{\\eta\_\{1\}^\{\\top\}\\eta\_\{2\}\}\{r^\{2\}\}\\right\)\.\(B\.42\)Using[EquationB\.10](https://arxiv.org/html/2610.06919#A2.E10)together with[EquationB\.34](https://arxiv.org/html/2610.06919#A2.E34),

dLS\(�1∥�2\)\\displaystyle d\_\{\\mathrm\{LS\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=1�​Ak​\(1�\)​\(1−�2​�​\(�1\)⊤​�​\(�2\)\)\\displaystyle=\\frac\{1\}\{\\tau\}A\_\{k\}\\left\(\\frac\{1\}\{\\tau\}\\right\)\\left\(1\-\\tau^\{2\}\\eta\(\\lambda\_\{1\}\)^\{\\top\}\\eta\(\\lambda\_\{2\}\)\\right\)\(B\.43\)=1�​Ak​\(1�\)​\(1−\(U⊤​�1\)⊤​\(U⊤​�2\)‖U⊤​�1‖2​‖U⊤​�2‖2\),\\displaystyle=\\frac\{1\}\{\\tau\}A\_\{k\}\\left\(\\frac\{1\}\{\\tau\}\\right\)\\left\(1\-\\frac\{\(U^\{\\top\}\\lambda\_\{1\}\)^\{\\top\}\(U^\{\\top\}\\lambda\_\{2\}\)\}\{\\\|U^\{\\top\}\\lambda\_\{1\}\\\|\_\{2\}\\,\\\|U^\{\\top\}\\lambda\_\{2\}\\\|\_\{2\}\}\\right\),\(B\.44\)which proves[EquationB\.35](https://arxiv.org/html/2610.06919#A2.E35)\. ∎

### B\.5Gaussian mixture model

Finally, we obtain a more expressive ambient\-space approximation by modeling the anchor distribution as a Gaussian mixture\. Take

z∈Rd,qZ=∑m=1M�m​𝒩​\(�m,�m\),T=Id,\\displaystyle z\\in\\mdmathbb R^\{d\},\\qquad q\_\{Z\}=\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\,\\mathcal\{N\}\(\\mu\_\{m\},\\Sigma\_\{m\}\),\\qquad T=\\mathrm\{Id\},\(B\.45\)where�m\>0\\pi\_\{m\}\>0,∑m=1M�m=1\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}=1, and�m≻0\\Sigma\_\{m\}\\succ 0\. Thus,z=z=\\gamma\. By the identity\-case construction in[SectionB\.1](https://arxiv.org/html/2610.06919#A2.SS1), the image\-side natural parameter is

�=��\.\\displaystyle\\eta=\\frac\{\\lambda\}\{\\tau\}\.\(B\.46\)This yields a multimodal ambient approximation to the population anchor geometry\.

###### Proposition 9\(Gaussian mixture anchor model\)\.

For the Gaussian mixture anchor model,

dGMM\(�1∥�2\)=B�GMM\(�2�∥�1�\),d\_\{\\mathrm\{GMM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=B\_\{\\kappa\_\{\\mathrm\{GMM\}\}\}\\\!\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\),\(B\.47\)where

�GMM\(�\)=log∑m=1M�mexp\(�m⊤�\+12�⊤�m�\)\.\\kappa\_\{\\mathrm\{GMM\}\}\(\\eta\)=\\log\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\exp\\\!\\left\(\\mu\_\{m\}^\{\\top\}\\eta\+\\frac\{1\}\{2\}\\eta^\{\\top\}\\Sigma\_\{m\}\\eta\\right\)\.\(B\.48\)

###### Proof\.

Starting from[EquationB\.45](https://arxiv.org/html/2610.06919#A2.E45),

�Z​\(�\)\\displaystyle\\kappa\_\{Z\}\(\\eta\)=log∫exp\(�⊤z\)qZ\(dz\)\\displaystyle=\\log\\int\\exp\(\\eta^\{\\top\}z\)\\,q\_\{Z\}\(dz\)\(B\.49\)=log∑m=1M�m∫exp\(�⊤z\)𝒩\(dz;�m,�m\)\.\\displaystyle=\\log\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\int\\exp\(\\eta^\{\\top\}z\)\\,\\mathcal\{N\}\(dz;\\mu\_\{m\},\\Sigma\_\{m\}\)\.\(B\.50\)Each inner integral is the Gaussian MGF, so

�Z\(�\)=log∑m=1M�mexp\(�m⊤�\+12�⊤�m�\),\\displaystyle\\kappa\_\{Z\}\(\\eta\)=\\log\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\exp\\\!\\left\(\\mu\_\{m\}^\{\\top\}\\eta\+\\frac\{1\}\{2\}\\eta^\{\\top\}\\Sigma\_\{m\}\\eta\\right\),\(B\.51\)which is exactly[EquationB\.48](https://arxiv.org/html/2610.06919#A2.E48)\. Hence,

dZ\(�1∥�2\)=B�GMM\(�2∥�1\)\.\\displaystyle d\_\{Z\}\(\\eta\_\{1\}\\\|\\eta\_\{2\}\)=B\_\{\\kappa\_\{\\mathrm\{GMM\}\}\}\(\\eta\_\{2\}\\\|\\eta\_\{1\}\)\.\(B\.52\)Using�​\(�\)=�/�\\eta\(\\lambda\)=\\lambda/\\tauin[EquationB\.10](https://arxiv.org/html/2610.06919#A2.E10),

dGMM\(�1∥�2\)\\displaystyle d\_\{\\mathrm\{GMM\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=dZ\(�1�∥�2�\)\\displaystyle=d\_\{Z\}\\\!\\left\(\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\right\)\(B\.53\)=B�GMM\(�2�∥�1�\),\\displaystyle=B\_\{\\kappa\_\{\\mathrm\{GMM\}\}\}\\\!\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\),\(B\.54\)which proves[EquationB\.47](https://arxiv.org/html/2610.06919#A2.E47)\. ∎

## Appendix CParameter estimation for structured anchor models

Given anchor embeddings\{\}ii=1n⊂Rd\\\{\{\}\_\{i\}\\\}\_\{i=1\}^\{n\}\\subset\\mdmathbb\{R\}^\{d\}, we estimate the parameters of each structured anchor model directly from the anchor set\.

##### Gaussian anchor model\.

For the Gaussian model, we fit the mean and covariance by maximum likelihood:

�^\\displaystyle\\hat\{\\mu\}=1n∑i=1ni,\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\{\}\_\{i\},\(C\.1\)�^\\displaystyle\\hat\{\\Sigma\}=1n∑i=1n\(−i�^\)\(−i�^\)⊤\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\{\}\_\{i\}\-\\hat\{\\mu\}\)\(\{\}\_\{i\}\-\\hat\{\\mu\}\)^\{\\top\}\.\(C\.2\)The fitted semantic divergence is obtained by substituting�^\\hat\{\\Sigma\}into[EquationB\.13](https://arxiv.org/html/2610.06919#A2.E13)\.

##### Low\-rank models\.

For the latent Gaussian and latent spherical models, we estimate the semantic subspaceU∈Rd×kU\\in\\mdmathbb\{R\}^\{d\\times k\}by the top\-kkeigenvectors of the anchor covariance matrix

�^=1n∑i=1n\(−i¯\)\(−i¯\)⊤,¯=1n∑i=1n\.i\\displaystyle\\hat\{\\Sigma\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\{\}\_\{i\}\-\\bar\{\\gamma\}\)\(\{\}\_\{i\}\-\\bar\{\\gamma\}\)^\{\\top\},\\qquad\\bar\{\\gamma\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\{\}\_\{i\}\.\(C\.3\)We choosekkto explain95%95\\%of the anchor covariance\. The fitted divergences are then obtained by substitutingU^\\hat\{U\}into[EquationB\.24](https://arxiv.org/html/2610.06919#A2.E24)and[EquationB\.35](https://arxiv.org/html/2610.06919#A2.E35), respectively\.

##### Gaussian mixture model\.

For the Gaussian mixture model, we fit the mixture weights, means, and covariances,

\{\(�m,�m,�m\)\}m=1M,\\displaystyle\\\{\(\\pi\_\{m\},\\mu\_\{m\},\\Sigma\_\{m\}\)\\\}\_\{m=1\}^\{M\},by expectation–maximization \(EM\) on the anchor embeddings\. The fitted parameters are then substituted into[EquationB\.48](https://arxiv.org/html/2610.06919#A2.E48)and[EquationB\.47](https://arxiv.org/html/2610.06919#A2.E47)\.

## Appendix DModel\-specific posterior and posterior\-like updates for posterior anchor modeling

This appendix gives model\-specific posterior and posterior\-like updates for the posterior anchor modeling framework introduced in[Section4](https://arxiv.org/html/2610.06919#S4)\. Broad anchors provide prior semantic structure for a parametric anchor model, precise task\-aligned anchors refine that structure, and the resulting model summary induces an adapted geometry throughB��^B\_\{\\kappa\_\{\\hat\{\\theta\}\}\}\.

### D\.1General adaptation framework

Suppose we are given a broad anchor set

�broad=\{,1broad…,\}mbroad\\displaystyle\\Gamma\_\{\\mathrm\{broad\}\}=\\\{\{\}^\{\\mathrm\{broad\}\}\_\{1\},\\dots,\{\}^\{\\mathrm\{broad\}\}\_\{m\}\\\}and a smaller set of precise task\-aligned anchors

�target=\{,1target…,\}ntarget\.\\displaystyle\\Gamma\_\{\\mathrm\{target\}\}=\\\{\{\}^\{\\mathrm\{target\}\}\_\{1\},\\dots,\{\}^\{\\mathrm\{target\}\}\_\{n\}\\\}\.Let\{q�:�∈�\}\\\{q\_\{\\theta\}:\\theta\\in\\Theta\\\}be a parametric family of anchor distributions together with model\-specific prior structure on�\\Theta\. A generic adaptation procedure is:

1. 1\.fit the parametric family on�broad\\Gamma\_\{\\mathrm\{broad\}\}to obtain a broad\-anchor estimate�broad\\theta\_\{\\mathrm\{broad\}\};
2. 2\.choose hyperparameters or regularization structure=⁡\(�broad\)\\Omega=\\Omega\(\\theta\_\{\\mathrm\{broad\}\}\)so that the update is centered around the broad semantic structure encoded by�broad\\theta\_\{\\mathrm\{broad\}\};
3. 3\.combine this prior structure with the precise anchors to obtain a model\-specific posterior or posterior\-like update over�\\theta;
4. 4\.use a model\-specific summary�^\\hat\{\\theta\}to define the adapted anchor modelq�^q\_\{\\hat\{\\theta\}\};
5. 5\.induce the adapted geometry through ��^\(�\)=log∫exp\(�⊤\)q�^\(d\),d�^\(�1∥�2\)=B��^\(�2�∥�1�\)\.\\displaystyle\\kappa\_\{\\hat\{\\theta\}\}\(\\xi\)=\\log\\int\\exp\(\\xi^\{\\top\}\\gamma\)\\,q\_\{\\hat\{\\theta\}\}\(d\\gamma\),\\qquad d\_\{\\hat\{\\theta\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=B\_\{\\kappa\_\{\\hat\{\\theta\}\}\}\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\)\.

We now specialize this framework to the model families used in the paper\.

### D\.2Gaussian anchor model with Normal–Inverse\-Wishart update

For the Gaussian anchor model,

q�​\(\)=𝒩⁡\(,�,�\),�=\(�,�\),�≻0\.\\displaystyle q\_\{\\theta\}\(\\gamma\)=\\mathcal\{N\}\(\\gamma;\\mu,\\Sigma\),\\qquad\\theta=\(\\mu,\\Sigma\),\\qquad\\Sigma\\succ 0\.As shown in[SectionB\.2](https://arxiv.org/html/2610.06919#A2.SS2), the induced geometry is

dG\(�1∥�2\)=12​�2\(�1−�2\)⊤�\(�1−�2\),\\displaystyle d\_\{\\mathrm\{G\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}\\Sigma\(\\lambda\_\{1\}\-\\lambda\_\{2\}\),so adaptation corresponds to updating the covariance structure of the anchor distribution\.

We place a Normal–Inverse\-Wishart prior on\(�,�\)\(\\mu,\\Sigma\):

�\\displaystyle\\Sigma∼ℐ𝒲\(,0�0\),\\displaystyle\\sim\\mathcal\{IW\}\(\{\}\_\{0\},\\nu\_\{0\}\),�\|�\\displaystyle\\mu\\penalty\\ \|\\penalty\\ \\Sigma∼𝒩⁡\(�0,1�0​�\)\.\\displaystyle\\sim\\mathcal\{N\}\\\!\\left\(\\mu\_\{0\},\\frac\{1\}\{\\kappa\_\{0\}\}\\Sigma\\right\)\.
The adaptation steps are:

1. 1\.Fit the broad Gaussian\.Fit a Gaussian model to the broad anchors: ¯broad\\displaystyle\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\}=1m∑i=1mibroad,\\displaystyle=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\{\}\_\{i\}^\{\\mathrm\{broad\}\},Sbroad\\displaystyle S\_\{\\mathrm\{broad\}\}=∑i=1m\(−broadi¯broad\)\(−broadi¯broad\)⊤\.\\displaystyle=\\sum\_\{i=1\}^\{m\}\(\{\}\_\{i\}^\{\\mathrm\{broad\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\}\)\(\{\}\_\{i\}^\{\\mathrm\{broad\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\}\)^\{\\top\}\.This gives the broad\-anchor estimate\(�broad,�broad\)\(\\mu\_\{\\mathrm\{broad\}\},\\Sigma\_\{\\mathrm\{broad\}\}\)\.
2. 2\.Initialize the prior\.Use the broad\-anchor estimate in an empirical\-Bayes step: �0\\displaystyle\\mu\_\{0\}=¯broad,\\displaystyle=\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\},0=\(�0−d−1\)​�broad,\\displaystyle=\(\\nu\_\{0\}\-d\-1\)\\Sigma\_\{\\mathrm\{broad\}\},where�0\\kappa\_\{0\}and�0\\nu\_\{0\}control prior strength\.
3. 3\.Update with precise anchors\.Given precise anchors�target\\Gamma\_\{\\mathrm\{target\}\}, compute ¯target\\displaystyle\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}=1n∑i=1nitarget,\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\{\}\_\{i\}^\{\\mathrm\{target\}\},Starget\\displaystyle S\_\{\\mathrm\{target\}\}=∑i=1n\(−targeti¯target\)\(−targeti¯target\)⊤\.\\displaystyle=\\sum\_\{i=1\}^\{n\}\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\)\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\)^\{\\top\}\.The posterior is again NIW with parameters �n\\displaystyle\\kappa\_\{n\}=�0\+n,\\displaystyle=\\kappa\_\{0\}\+n,�n\\displaystyle\\nu\_\{n\}=�0\+n,\\displaystyle=\\nu\_\{0\}\+n,�n\\displaystyle\\mu\_\{n\}=�0�0\+n¯target�0\+n,\\displaystyle=\\frac\{\\kappa\_\{0\}\\mu\_\{0\}\+n\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\}\{\\kappa\_\{0\}\+n\},n=\+0Starget\+�0​n�0\+n\(¯target−�0\)\(¯target−�0\)⊤\.\\displaystyle=\{\}\_\{0\}\+S\_\{\\mathrm\{target\}\}\+\\frac\{\\kappa\_\{0\}n\}\{\\kappa\_\{0\}\+n\}\(\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\-\\mu\_\{0\}\)\(\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\-\\mu\_\{0\}\)^\{\\top\}\.
4. 4\.Form the adapted model\.Use a posterior summary\(�^,�^\)\(\\hat\{\\mu\},\\hat\{\\Sigma\}\), such as the posterior mean or MAP estimate, to define the adapted Gaussian anchor model q�^​\(\)=𝒩⁡\(,�^,�^\)\.\\displaystyle q\_\{\\hat\{\\theta\}\}\(\\gamma\)=\\mathcal\{N\}\(\\gamma;\\hat\{\\mu\},\\hat\{\\Sigma\}\)\.
5. 5\.Induce the adapted geometry\.The adapted geometry is dGadapt\(�1∥�2\)=12​�2\(�1−�2\)⊤�^\(�1−�2\)\.\\displaystyle d\_\{\\mathrm\{G\}\}^\{\\mathrm\{adapt\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}\\hat\{\\Sigma\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)\.

### D\.3Latent Gaussian model with matrix Bingham update

For the latent Gaussian model, the anchor distribution is determined by a low\-dimensional semantic subspace\. Concretely, we use the ambient surrogate

q�​\(\)=𝒩⁡\(,�,U​U⊤\+"​I\),�=\(�,U\),\\displaystyle q\_\{\\theta\}\(\\gamma\)=\\mathcal\{N\}\(\\gamma;\\mu,UU^\{\\top\}\+\\varepsilon I\),\\qquad\\theta=\(\\mu,U\),whereU∈Rd×kU\\in\\mdmathbb R^\{d\\times k\}has orthonormal columns,�∈Rd\\mu\\in\\mdmathbb R^\{d\}, and"\>0\\varepsilon\>0is a small isotropic regularization used in implementation\. As shown in[SectionB\.3](https://arxiv.org/html/2610.06919#A2.SS3):

dLG\(�1∥�2\)=12​�2\(�1−�2\)⊤UU⊤\(�1−�2\)\.\\displaystyle d\_\{\\mathrm\{LG\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}UU^\{\\top\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)\.Thus, adaptation amounts to updating the subspaceUU\.

We place a matrix Bingham prior onUUover the Stiefel manifold

St⁡\(d,k\)=\{U∈Rd×k:U⊤​U=Ik\},\\displaystyle\\mathrm\{St\}\(d,k\)=\\\{U\\in\\mdmathbb R^\{d\\times k\}:U^\{\\top\}U=I\_\{k\}\\\},with density

p⁡\(U\|A\)∝exp⁡\(tr⁡\(U⊤​A​U\)\)\.\\displaystyle p\(U\\penalty\\ \|\\penalty\\ A\)\\propto\\exp\\\!\\bigl\(\\operatorname\{tr\}\(U^\{\\top\}AU\)\\bigr\)\.Under the Gaussian covariance model above, this prior is conjugate for the subspace parameterUU\.

The adaptation steps are:

1. 1\.Fit the broad semantic subspace\.Compute the empirical covariance of the broad anchors and letUbroadU\_\{\\mathrm\{broad\}\}be its top\-kkeigenvectors: Cov^\(�broad\)=1m∑i=1m\(−broadi¯broad\)\(−broadi¯broad\)⊤\.\\displaystyle\\widehat\{\\operatorname\{Cov\}\}\(\\Gamma\_\{\\mathrm\{broad\}\}\)=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\(\{\}\_\{i\}^\{\\mathrm\{broad\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\}\)\(\{\}\_\{i\}^\{\\mathrm\{broad\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{broad\}\}\)^\{\\top\}\.
2. 2\.Initialize the Bingham prior\.Use the broad subspace to define A=�​Ubroad​Ubroad⊤,\\displaystyle A=\\alpha\\,U\_\{\\mathrm\{broad\}\}U\_\{\\mathrm\{broad\}\}^\{\\top\},where�\>0\\alpha\>0controls prior strength\.
3. 3\.Update with precise anchors\.Let Cov^\(�target\)=1n∑i=1n\(−targeti¯target\)\(−targeti¯target\)⊤\.\\displaystyle\\widehat\{\\operatorname\{Cov\}\}\(\\Gamma\_\{\\mathrm\{target\}\}\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\)\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{\\mathrm\{target\}\}\)^\{\\top\}\.Under the Gaussian covariance model, the posterior overUUremains in matrix\-Bingham form, with updated parameter matrix Apost=A\+�​Cov^​\(�target\),\\displaystyle A\_\{\\mathrm\{post\}\}=A\+\\beta\\,\\widehat\{\\operatorname\{Cov\}\}\(\\Gamma\_\{\\mathrm\{target\}\}\),where �=n2​\(1"−11\+"\)\.\\displaystyle\\beta=\\frac\{n\}\{2\}\\left\(\\frac\{1\}\{\\varepsilon\}\-\\frac\{1\}\{1\+\\varepsilon\}\\right\)\.
4. 4\.Form the adapted model via MAP\.We use the MAP estimator under the Bingham posterior: Upost=arg⁡maxU∈St⁡\(d,k\)​tr⁡\(U⊤​Apost​U\),\\displaystyle U\_\{\\mathrm\{post\}\}=\\arg\\max\_\{U\\in\\mathrm\{St\}\(d,k\)\}\\operatorname\{tr\}\(U^\{\\top\}A\_\{\\mathrm\{post\}\}U\),which is given by the top\-kkeigenvectors ofApostA\_\{\\mathrm\{post\}\}\.
5. 5\.Induce the adapted geometry\.The adapted geometry is dLGadapt\(�1∥�2\)=12​�2\(�1−�2\)⊤UpostUpost⊤\(�1−�2\)\.\\displaystyle d\_\{\\mathrm\{LG\}\}^\{\\mathrm\{adapt\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=\\frac\{1\}\{2\\tau^\{2\}\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)^\{\\top\}U\_\{\\mathrm\{post\}\}U\_\{\\mathrm\{post\}\}^\{\\top\}\(\\lambda\_\{1\}\-\\lambda\_\{2\}\)\.

### D\.4Latent spherical model with posterior\-like Bingham update

For the latent spherical model, adaptation again reduces to updating the semantic subspaceUU\. We use the same matrix\-Bingham form as a tractable posterior\-like update, driven by the broad\-anchor subspace and the precise\-anchor covariance\. As in the latent Gaussian case, we initialize

A=�​Ubroad​Ubroad⊤\\displaystyle A=\\alpha\\,U\_\{\\mathrm\{broad\}\}U\_\{\\mathrm\{broad\}\}^\{\\top\}from the broad\-anchor subspaceUbroadU\_\{\\mathrm\{broad\}\}, then combine this with the precise\-anchor covariance to form

Apost=A\+�​Cov^​\(�target\),\\displaystyle A\_\{\\mathrm\{post\}\}=A\+\\beta\\,\\widehat\{\\operatorname\{Cov\}\}\(\\Gamma\_\{\\mathrm\{target\}\}\),where�\>0\\beta\>0controls the influence of the precise anchors\. The adapted subspace is the MAP estimate

Upost=arg⁡maxU∈St⁡\(d,k\)​tr⁡\(U⊤​Apost​U\),\\displaystyle U\_\{\\mathrm\{post\}\}=\\arg\\max\_\{U\\in\\mathrm\{St\}\(d,k\)\}\\operatorname\{tr\}\(U^\{\\top\}A\_\{\\mathrm\{post\}\}U\),given by the top\-kkeigenvectors ofApostA\_\{\\mathrm\{post\}\}\. The adapted geometry is obtained by substitutingUpostU\_\{\\mathrm\{post\}\}into the latent spherical divergence\.

### D\.5Gaussian mixture model with Dirichlet–NIW update

For the Gaussian mixture model, we parameterize the anchor distribution as

q�​\(d​\)=∑m=1M�m​𝒩​\(d​,�m,�m\),�=\{\(�m,�m,�m\)\}m=1M,\\displaystyle q\_\{\\theta\}\(d\\gamma\)=\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\,\\mathcal\{N\}\(d\\gamma;\\mu\_\{m\},\\Sigma\_\{m\}\),\\qquad\\theta=\\\{\(\\pi\_\{m\},\\mu\_\{m\},\\Sigma\_\{m\}\)\\\}\_\{m=1\}^\{M\},where�m\>0\\pi\_\{m\}\>0,∑m=1M�m=1\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}=1, and�m≻0\\Sigma\_\{m\}\\succ 0\. The induced geometry is the Bregman divergence generated by

�GMM\(�\)=log∑m=1M�mexp\(�m⊤�\+12�⊤�m�\)\.\\displaystyle\\kappa\_\{\\mathrm\{GMM\}\}\(\\xi\)=\\log\\sum\_\{m=1\}^\{M\}\\pi\_\{m\}\\exp\\\!\\left\(\\mu\_\{m\}^\{\\top\}\\xi\+\\frac\{1\}\{2\}\\xi^\{\\top\}\\Sigma\_\{m\}\\xi\\right\)\.Unlike the single\-Gaussian case, this geometry is generally nonquadratic and can represent multiple semantic modes\.

We place a symmetric Dirichlet prior on the mixture weights and independent Normal–Inverse\-Wishart priors on the component parameters:

\(�1,…,�M\)\\displaystyle\(\\pi\_\{1\},\\dots,\\pi\_\{M\}\)∼Dir⁡\(�,…,�\),\\displaystyle\\sim\\operatorname\{Dir\}\(\\alpha,\\dots,\\alpha\),�m\\displaystyle\\Sigma\_\{m\}∼ℐ𝒲\(,0,m�0,m\),\\displaystyle\\sim\\mathcal\{IW\}\(\{\}\_\{0,m\},\\nu\_\{0,m\}\),�m\|�m\\displaystyle\\mu\_\{m\}\\penalty\\ \|\\penalty\\ \\Sigma\_\{m\}∼𝒩\(�0,m,1�0,m�m\),m=1,…,M\.\\displaystyle\\sim\\mathcal\{N\}\\\!\\left\(\\mu\_\{0,m\},\\frac\{1\}\{\\kappa\_\{0,m\}\}\\Sigma\_\{m\}\\right\),\\qquad m=1,\\dots,M\.
The adaptation steps are:

1. 1\.Fit the broad mixture\.Fit anMM\-component GMM to the broad anchors, producing \{�mbroad,�mbroad,�mbroad\}m=1M\.\\displaystyle\\left\\\{\\pi\_\{m\}^\{\\mathrm\{broad\}\},\\mu\_\{m\}^\{\\mathrm\{broad\}\},\\Sigma\_\{m\}^\{\\mathrm\{broad\}\}\\right\\\}\_\{m=1\}^\{M\}\.
2. 2\.Initialize the prior\.Use the broad\-anchor fit to set the component\-specific hyperparameters �0,m\\displaystyle\\mu\_\{0,m\}=�mbroad,\\displaystyle=\\mu\_\{m\}^\{\\mathrm\{broad\}\},0,m=\(�0,m−d−1\)​�mbroad,\\displaystyle=\(\\nu\_\{0,m\}\-d\-1\)\\Sigma\_\{m\}^\{\\mathrm\{broad\}\},while resetting the mixture weights to a symmetric prior \(�1,…,�M\)∼Dir⁡\(�,…,�\)\.\\displaystyle\(\\pi\_\{1\},\\dots,\\pi\_\{M\}\)\\sim\\operatorname\{Dir\}\(\\alpha,\\dots,\\alpha\)\.Thus, the broad anchors determine the semantic modes and local covariance scales, but not their prior prevalence\.
3. 3\.Update with precise anchors using soft assignments\.Given precise anchors�target\\Gamma\_\{\\mathrm\{target\}\}, compute responsibilities ri​m=p\(m\|\)targeti\\displaystyle r\_\{im\}=p\(m\\penalty\\ \|\\penalty\\ \{\}\_\{i\}^\{\\mathrm\{target\}\}\)under the current mixture model\. Soft assignments allow precise anchors to refine multiple broad semantic modes simultaneously\. Define weighted sufficient statistics Nm\\displaystyle N\_\{m\}=∑i=1nri​m,\\displaystyle=\\sum\_\{i=1\}^\{n\}r\_\{im\},¯m\\displaystyle\\bar\{\\gamma\}\_\{m\}=1Nm∑i=1nri​mitarget,\\displaystyle=\\frac\{1\}\{N\_\{m\}\}\\sum\_\{i=1\}^\{n\}r\_\{im\}\{\}\_\{i\}^\{\\mathrm\{target\}\},Sm\\displaystyle S\_\{m\}=∑i=1nri​m\(−targeti¯m\)\(−targeti¯m\)⊤\.\\displaystyle=\\sum\_\{i=1\}^\{n\}r\_\{im\}\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{m\}\)\(\{\}\_\{i\}^\{\\mathrm\{target\}\}\-\\bar\{\\gamma\}\_\{m\}\)^\{\\top\}\.The updated hyperparameters are �m′\\displaystyle\\alpha\_\{m\}^\{\\prime\}=�\+Nm,\\displaystyle=\\alpha\+N\_\{m\},�m′\\displaystyle\\kappa\_\{m\}^\{\\prime\}=�0,m\+Nm,\\displaystyle=\\kappa\_\{0,m\}\+N\_\{m\},�m′\\displaystyle\\nu\_\{m\}^\{\\prime\}=�0,m\+Nm,\\displaystyle=\\nu\_\{0,m\}\+N\_\{m\},�m′\\displaystyle\\mu\_\{m\}^\{\\prime\}=�0,m�0,m\+Nm¯m�0,m\+Nm,\\displaystyle=\\frac\{\\kappa\_\{0,m\}\\mu\_\{0,m\}\+N\_\{m\}\\bar\{\\gamma\}\_\{m\}\}\{\\kappa\_\{0,m\}\+N\_\{m\}\},′m\\displaystyle\{\}\_\{m\}^\{\\prime\}=\+0,mSm\+�0,m​Nm�0,m\+Nm\(¯m−�0,m\)\(¯m−�0,m\)⊤\.\\displaystyle=\{\}\_\{0,m\}\+S\_\{m\}\+\\frac\{\\kappa\_\{0,m\}N\_\{m\}\}\{\\kappa\_\{0,m\}\+N\_\{m\}\}\(\\bar\{\\gamma\}\_\{m\}\-\\mu\_\{0,m\}\)\(\\bar\{\\gamma\}\_\{m\}\-\\mu\_\{0,m\}\)^\{\\top\}\.
4. 4\.Form the adapted model\.Use posterior means to define the adapted mixture parameters �^m\\displaystyle\\hat\{\\pi\}\_\{m\}=�m′∑ℓ=1M�ℓ′,\\displaystyle=\\frac\{\\alpha\_\{m\}^\{\\prime\}\}\{\\sum\_\{\\ell=1\}^\{M\}\\alpha\_\{\\ell\}^\{\\prime\}\},�^m\\displaystyle\\hat\{\\mu\}\_\{m\}=�m′,\\displaystyle=\\mu\_\{m\}^\{\\prime\},�^m\\displaystyle\\hat\{\\Sigma\}\_\{m\}=′m�m′−d−1,\\displaystyle=\\frac\{\{\}\_\{m\}^\{\\prime\}\}\{\\nu\_\{m\}^\{\\prime\}\-d\-1\},provided�m′\>d\+1\\nu\_\{m\}^\{\\prime\}\>d\+1\. This gives the adapted anchor model q�^​\(d​\)=∑m=1M�^m​𝒩​\(d​,�^m,�^m\)\.\\displaystyle q\_\{\\hat\{\\theta\}\}\(d\\gamma\)=\\sum\_\{m=1\}^\{M\}\\hat\{\\pi\}\_\{m\}\\,\\mathcal\{N\}\(d\\gamma;\\hat\{\\mu\}\_\{m\},\\hat\{\\Sigma\}\_\{m\}\)\.
5. 5\.Induce the adapted geometry\.The adapted geometry is dGMMadapt\(�1∥�2\)=B��^\(�2�∥�1�\),\\displaystyle d\_\{\\mathrm\{GMM\}\}^\{\\mathrm\{adapt\}\}\(\\lambda\_\{1\}\\\|\\lambda\_\{2\}\)=B\_\{\\kappa\_\{\\hat\{\\theta\}\}\}\\\!\\left\(\\frac\{\\lambda\_\{2\}\}\{\\tau\}\\Big\\\|\\frac\{\\lambda\_\{1\}\}\{\\tau\}\\right\),where ��^\(�\)=log∑m=1M�^mexp\(�^m⊤�\+12�⊤�^m�\)\.\\displaystyle\\kappa\_\{\\hat\{\\theta\}\}\(\\xi\)=\\log\\sum\_\{m=1\}^\{M\}\\hat\{\\pi\}\_\{m\}\\exp\\\!\\left\(\\hat\{\\mu\}\_\{m\}^\{\\top\}\\xi\+\\frac\{1\}\{2\}\\xi^\{\\top\}\\hat\{\\Sigma\}\_\{m\}\\xi\\right\)\.

### D\.6Discussion

These model\-specific updates instantiate the same principle: posterior anchor modeling updates a parametric anchor model using broad anchors as prior semantic structure and precise anchors as task\-specific evidence, which in turn updates the induced Bregman geometry in image space\. The models differ in the semantic structure they can transfer: a single global covariance in the Gaussian case, a shared low\-dimensional semantic subspace in the low\-rank case, and multiple reweighted semantic modes in the mixture case\. This is precisely why richer models such as the GMM can support forms of semantic transfer that would not arise from simple projection heuristics\.

## Appendix EImplementation details

### E\.1Datasets

We evaluate on ImageNet, ImageNet\-R, CUB\-200, Sketchy, and CheXpert\.

##### ImageNet\.

We use the standard training split as the retrieval gallery/keys and the validation split as queries\. Relevance is defined by class agreement\. ImageNet class\-name anchors are formed from the class labels using the template “a photo of a \{x\}”\.

##### ImageNet\-R\.

Queries and keys are both drawn from ImageNet\-R, with each query removed from its own key set\. Object relevance is defined by ImageNet class agreement, while style labels for the style\-control experiments are parsed from the ImageNet\-R filenames\. Object\-name anchors use the template “a photo of a \{x\}”, and style anchors use “a \{s\} rendition of an object”\. For style retrieval we discard images that have “misc” style labels, since these are not well\-defined styles and do not have corresponding anchors\.

##### CUB\-200\.

We use the standard train/test split, with training images as the keys and test images as queries\. Relevance is defined by bird\-class agreement\. CUB anchors are formed from the class names using the same class\-name template\.

##### Sketchy\.

We use the Sketchy Extended Split 2 protocol for zero\-shot sketch\-based image retrieval, following\[[Yel\+18](https://arxiv.org/html/2610.06919#bib.bibx37)\]\. The test set consists of 21 classes chosen to be absent from ImageNet\-1K\. We use sketches from these unseen classes as queries and photos from the same classes as the retrieval gallery; no training\-class images are used\. Relevance is defined by class agreement, and anchors are formed from the class names using the template “a photo of a \{x\}”\.

##### CheXpert\.

We evaluate multi\-label retrieval on CheXpert using pathology labels as relevance targets\. The official training split is used as keys and the validation split as queries\. We discard the training images with uncertain labels\. For each target pathology, we construct report\-style text anchors describing that pathology\. The exact LLM prompt is given in[SectionE\.6](https://arxiv.org/html/2610.06919#A5.SS6)\.

##### Synthetic collection for Figure 1\.

[Figure1](https://arxiv.org/html/2610.06919#S1.F1)uses a synthetic image collection in which dog breed, background, and clothing are systematically varied\. Images were generated using OpenAI’sgpt\-image\-2model\. The displayed neighbors are obtained directly by retrieval under cosine similarity and the corresponding anchor divergences\.

### E\.2Models

All natural\-image experiments use MetaCLIP\-2\. All image and text embeddings are normalized to unitℓ2\\ell\_\{2\}norm, and we use the model\-provided logit scale,exp⁡\(clip\_model\.logit\_scale\)\\exp\(\\texttt\{clip\\\_model\.logit\\\_scale\}\), as the inverse temperature\.

For CheXpert, we use CheXzero and report results averaged over the 10 released model weights\.

### E\.3Image\-text inversion baseline

As a retrieval baseline, we use image\-text inversion \(ITI\), which converts a query image into a text\-side proxy by selecting the anchor with highest image–text compatibility\. Given an anchor set�=\{,1…,\}N\\Gamma=\\\{\{\}\_\{1\},\\dots,\{\}\_\{N\}\\\}and a query image embedding�q\\lambda\_\{q\}, we first choose

i⋆\(�q\)=argmax1≤i≤N�q⊤\.i\\displaystyle i^\{\\star\}\(\\lambda\_\{q\}\)=\\arg\\max\_\{1\\leq i\\leq N\}\\;\\lambda\_\{q\}^\{\\top\}\{\}\_\{i\}\.\(E\.1\)We then take the corresponding anchor

~\(�q\)=i⋆​\(�q\)\\displaystyle\\tilde\{\\gamma\}\(\\lambda\_\{q\}\)=\{\}\_\{i^\{\\star\}\(\\lambda\_\{q\}\)\}\(E\.2\)as the inverted text representation of the query\. Each gallery image embedding�k\\lambda\_\{k\}is then scored by cosine similarity to this inferred text embedding:

sITI\(�q,�k\)=~\(�q\)⊤�k\.\\displaystyle s\_\{\\mathrm\{ITI\}\}\(\\lambda\_\{q\},\\lambda\_\{k\}\)=\\tilde\{\\gamma\}\(\\lambda\_\{q\}\)^\{\\top\}\\lambda\_\{k\}\.\(E\.3\)Keys are ranked in decreasing order ofsITI​\(�q,�k\)s\_\{\\mathrm\{ITI\}\}\(\\lambda\_\{q\},\\lambda\_\{k\}\)\.

Thus, ITI uses the anchor set only to select a single best\-matching text proxy for the query, after which retrieval is performed by standard image\-text cosine similarity\.

### E\.4Evaluation metric

We evaluate retrieval performance with the area under the precision–recall curve \(AUPRC\)\. For a given query, let

𝒢=\{x1,…,xN\}\\displaystyle\\mathcal\{G\}=\\\{x\_\{1\},\\dots,x\_\{N\}\\\}denote the key set, letsis\_\{i\}be the retrieval score assigned to key itemxix\_\{i\}, and let

yi∈\{0,1\}\\displaystyle y\_\{i\}\\in\\\{0,1\\\}indicate whetherxix\_\{i\}is semantically relevant to the query\. Sorting the keys by score induces a ranked list, and for any thresholdtton the score we define

Precision⁡\(t\)\\displaystyle\\operatorname\{Precision\}\(t\)=∑i=1N𝟏\{si≥t\}yi∑i=1N𝟏\{si≥t\},\\displaystyle=\\frac\{\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{s\_\{i\}\\geq t\\\}y\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{s\_\{i\}\\geq t\\\}\},\(E\.4\)Recall⁡\(t\)\\displaystyle\\operatorname\{Recall\}\(t\)=∑i=1N𝟏\{si≥t\}yi∑i=1Nyi\.\\displaystyle=\\frac\{\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{s\_\{i\}\\geq t\\\}y\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}y\_\{i\}\}\.\(E\.5\)The precision–recall curve is the set of points

\(Recall⁡\(t\),Precision⁡\(t\)\)\\displaystyle\\bigl\(\\operatorname\{Recall\}\(t\),\\operatorname\{Precision\}\(t\)\\bigr\)obtained as the threshold varies over the ranked list, and AUPRC is the area under this curve:

AUPRC=∫01Precision⁡\(r\)​𝑑r,\\displaystyle\\operatorname\{AUPRC\}=\\int\_\{0\}^\{1\}\\operatorname\{Precision\}\(r\)\\,dr,\(E\.6\)with the discrete implementation given by the corresponding finite\-sum approximation over ranked thresholds\.

### E\.5Parametric models and adaptation

##### Gaussian model\.

We fit the Gaussian anchor model by maximum likelihood on the anchor embeddings\.

##### Low\-rank models\.

For the latent Gaussian and latent spherical models, we estimate the semantic subspaceUUby the top\-kkeigenvectors of the centered anchor covariance matrix\. Unless otherwise specified, we choosekkto explain95%95\\%of the anchor covariance\. In the semantic adaptation experiments, where a fixed latent dimension is required, we usek=128k=128\. For the ambient surrogate used in latent Gaussian adaptation, we set the isotropic regularization to"=10−8\\varepsilon=10^\{\-8\}\.

##### Gaussian mixture model\.

We fit the Gaussian mixture model by expectation–maximization, with the number of mixture components specified in the corresponding plots\. EM is initialized withkk\-means, each covariance matrix is regularized by adding10−5​I10^\{\-5\}I, and optimization is terminated when the improvement in log\-likelihood falls below10−310^\{\-3\}or when 200 iterations are reached\.

##### Adaptation\.

Gaussian adaptation uses the Normal–Inverse\-Wishart update in[SectionD\.2](https://arxiv.org/html/2610.06919#A4.SS2)\. Latent Gaussian adaptation uses the matrix\-Bingham update in[SectionD\.3](https://arxiv.org/html/2610.06919#A4.SS3)\. Latent spherical adaptation uses the posterior\-like Bingham update in[SectionD\.4](https://arxiv.org/html/2610.06919#A4.SS4)\. Gaussian mixture adaptation uses the Dirichlet–NIW update in[SectionD\.5](https://arxiv.org/html/2610.06919#A4.SS5)\.

##### Adaptation hyperparameters\.

For Gaussian adaptation, we use�0=5\\kappa\_\{0\}=5and�0=d\+20\\nu\_\{0\}=d\+20\. The low\-rank Bingham updates have prior strength�=50\\alpha=50and"=10−8\\varepsilon=10^\{\-8\}\. For GMM adaptation, we use Dirichlet concentration�=1\\alpha=1,�0,m=5\\kappa\_\{0,m\}=5, and�0,m=d\+20\\nu\_\{0,m\}=d\+20, shared across mixture components unless otherwise specified\.

### E\.6CheXpert anchor generation

For CheXpert, we generate pathology\-specific report\-style anchors with ChatGPT 5\.4 Thinking\. The prompt asks for a Python dictionary whose keys are the CheXpert conditions and whose values are lists of possible doctor report notes for the corresponding condition\. Each list contains both positive and negative entries and is sized to contain roughly 50 strings\. The resulting pathology\-specific anchor lists are then embedded with CheXzero and used as the anchor sets defining the corresponding retrieval geometries\.

##### Prompt\.

> For the conditions in LABELS = \[“Enlarged Cardiomediastinum”, “Cardiomegaly”, “Lung Opacity”, “Lung Lesion”, “Edema”, “Consolidation”, “Pneumonia”, “Atelectasis”, “Pneumothorax”, “Pleural Effusion”, “Pleural Other”, “Fracture”, “Support Devices”\] Give a python dictionary where the keys are the different conditions and the values are a list of strings of different possible doctor report notes for that condition\. Have positive and negative entries\. Have each list be of size around 50

##### Usage in retrieval\.

For each target pathology, we extract the corresponding list from the generated dictionary, embed those report\-style strings with CheXzero, and use the resulting text embeddings as the pathology\-specific anchor set for retrieval\.

## Appendix FAdditional Experiments

### F\.1Parametric anchor models on CheXpert

The main text reports CheXpert results for the pathology\-specialized softmax anchor divergence\. Here we include the corresponding parametric anchor models on the same retrieval task\.[Table3](https://arxiv.org/html/2610.06919#A6.T3)shows that Gaussian anchor divergence performs comparably to softmax on average, while the latent models also improve over cosine, indicating that parametric anchor geometries can preserve clinically relevant coordinates beyond the natural\-image setting\.

Table 3:Parametric CheXpert anchor divergences\.Entries are pathology\-wise mean AUPRC differences relative to cosine similarity, with 95% confidence interval\. Results average over independently generated anchor sets and CheXzero checkpoints, with intervals accounting for bootstrap and anchor\-generation variability\. Softmax and Gaussian anchor divergences give the largest average gains, while latent models also improve over cosine on average\.
### F\.2Parametric models across anchor budgets on additional datasets

The main text studies how empirical and parametric anchor models behave as the number of available anchors varies on ImageNet\. We observe the same qualitative tradeoff on ImageNet\-R and CUB\-200\. As shown in[Figure6](https://arxiv.org/html/2610.06919#A6.F6), parametric anchor divergences are strongest over much of the limited\- and moderate\-anchor regime, while softmax improves steadily with anchor coverage and becomes strongest when the full class\-anchor set is available\. Thus, the anchor\-coverage tradeoff observed on ImageNet is not specific to that dataset\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/other_datasets_auprc_vs_num_precise_combined.png)Figure 6:Parametric structure is useful when anchor coverage is limited\.AUPRC as the number of class\-name anchors varies onImageNet\-R\(left\) andCUB\-200\(right\)\. Parametric anchor divergences are strongest across much of the limited\- and moderate\-anchor regime, while softmax improves with anchor coverage and is strongest when the full class\-anchor set is available\.
### F\.3Exploratory model\-complexity diagnostic

We include an exploratory diagnostic on how mixture complexity interacts with anchor\-distribution structure\. In the anchor\-distribution view, parametric models are structured approximations toqq: increasing flexibility can reduce approximation bias when the anchor distribution is heterogeneous, but can also introduce unnecessary complexity or estimation noise when simpler structure is sufficient\.

We compare ImageNet and CUB\-200 as illustrative regimes\. ImageNet class\-name anchors span many coarse semantic groups, while CUB\-200 anchors are narrower and concentrated around fine\-grained bird categories\.[Figure7](https://arxiv.org/html/2610.06919#A6.F7)shows different trends across these datasets: increasing the number of mixture components tends to help on ImageNet, while gains are smaller and less consistent on CUB\-200\. This diagnostic is not intended as a complete model\-selection study, but illustrates that parametric anchor models introduce structure whose usefulness depends on the structure of the anchor set\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/imagenet_cub_gmm_analysis.png)Figure 7:Mixture complexity behaves differently across anchor sets\.On ImageNet, increasing the number of GMM components generally improves retrieval, consistent with a heterogeneous class\-name anchor distribution\. On CUB\-200, gains are smaller and less consistent, suggesting that simpler structured geometries may already capture much of the relevant anchor variation\.
### F\.4Computational scaling with anchor\-set size

Parametric anchor models also provide a computational advantage when the number of observed anchors is large\. LetQQdenote the number of queries,GGthe number of gallery items,NNthe number of anchors, andddthe embedding dimension\. A direct implementation of softmax anchor divergence constructs anNN\-dimensional distribution for each image, giving computational costO⁡\(Q​G​N\+\(Q\+G\)​N​d\)O\(QGN\+\(Q\+G\)Nd\)\.

For the symmetrized divergence used in our experiments, the subsequent query–gallery comparison can be computed exactly from cacheddd\-dimensional statistics\. Writing�=�/�\\eta=\\lambda/\\tauand�​\(�\)=∇�q^�​\(�\)\\mu\(\\lambda\)=\\nabla\\kappa\_\{\\hat\{q\}\_\{\\Gamma\}\}\(\\eta\), we have

dsym​\(�q,�g\)=12​\(�q−�g\)⊤​\(�​\(�q\)−�​\(�g\)\)\.d\_\{\\mathrm\{sym\}\}\(\\lambda\_\{q\},\\lambda\_\{g\}\)=\\frac\{1\}\{2\}\(\\eta\_\{q\}\-\\eta\_\{g\}\)^\{\\top\}\\bigl\(\\mu\(\\lambda\_\{q\}\)\-\\mu\(\\lambda\_\{g\}\)\\bigr\)\.\(F\.1\)Thus, after caching�​\(�\)\\mu\(\\lambda\), softmax anchor divergence has total cost

O⁡\(\(Q\+G\)​N​d\+Q​G​d\),O\(\(Q\+G\)Nd\+QGd\),so dependence onNNremains in the gallery and query dependent preprocessing step\. In contrast, a fitted parametric anchor model summarizes the observed anchors by a fixed set of parameters\. Once this fitting step is complete, query and gallery preprocessing and pairwise evaluation need not depend on the original anchor\-set size\.

Table 4:Runtime scaling with anchor\-set size\.Wall\-clock retrieval time \(in seconds\) on a single NVIDIA H100 NVL GPU for 10,000 queries and 50,000 gallery images\. Runtimes include fitting, preprocessing, and retrieval and are averaged over 10 runs following 3 warm\-up runs\.
We examine this empirically by varying the number of anchors and timing retrieval for 10,000 queries against 50,000 gallery image embeddings on a single NVIDIA H100 NVL GPU\. Runtimes include model fitting, and retrieval, and are averaged over 10 runs following 3 warm\-up runs\. For softmax anchor divergence, anchors are streamed in chunks when computing the cached statistics, so additional peak GPU memory does not grow substantially withNN\.[Table4](https://arxiv.org/html/2610.06919#A6.T4)shows that softmax runtime increases with anchor\-set size as anchor\-dependent preprocessing becomes appreciable, whereas Gaussian anchor divergence remains nearly constant after the anchor distribution has been fit\.

### F\.5Qualitative object–style retrieval examples

[Figure8](https://arxiv.org/html/2610.06919#A6.F8)shows additional object\- and style\-anchor retrieval examples\. Object anchors retrieve images with the same object across styles, while style anchors retrieve images with the same style across object categories\. This pattern appears for both empirical softmax and parametric anchor divergences, suggesting that parametric models preserve the semantic coordinate selected by the anchors\.

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/multi_match_neurips_10134_labeled.png)

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/multi_match_neurips_23049_unlabeled.png)

![Refer to caption](https://arxiv.org/html/2610.06919v1/figures/multi_match_neurips_11160_unlabeled.png)

Figure 8:Qualitative examples of anchor\-controlled retrieval\.Blue boxes show object\-anchor retrieval, which prioritizes object identity across styles; red boxes show style\-anchor retrieval, which prioritizes style across object categories\. Rows compare some of the different anchor divergences\.