Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests

# Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests
Source: [https://arxiv.org/html/2608.06908](https://arxiv.org/html/2608.06908)
###### Abstract

We propose Zero\-phase Component Analysis \(ZCA\) whitening as a geometric pre\-processing step for the Word Embedding Association Test \(WEAT\)\. WEAT is a bias measurement method widely used in both computational social science and AI fairness research\. It relies on cosine similarity as a measure of semantic association, which assumes that the embedding space is approximately isotropic\. However, prior work has reported that many widely used language models do not satisfy this assumption, raising concerns about the reliability of bias measurements\. ZCA whitening transforms the covariance of the embedding space into the identity matrix while minimizing perturbation to the original vectors\. This transformation restores the isotropy condition on which WEAT relies\. We evaluate our approach on ten standard WEAT test suites and seven models spanning three architectural families, yielding 70 model–task combinations\. The results show that ZCA whitening substantially reduces the anisotropy of the embedding spaces across all models\. Particularly for highly anisotropic models, we further observe improvements on standard semantic similarity benchmarks, indicating that the calibrated space better captures semantic associations\. After calibration, over 30% of WEAT results change significance status, and effect sizes shift in both directions depending on bias category\. These shifts suggest that uncalibrated measurements may both overestimate and underestimate the associations encoded in the embedding space\. These findings indicate that previously reported bias measurements in anisotropic embedding spaces should be interpreted with caution and may benefit from re\-evaluation with calibrated methods\. Our approach contributes to restoring the measurement foundation of WEAT across both computational social science and AI fairness research\.

Code—https://github\.com/seigit/zca\-weat

Accepted at the 9th AAAI/ACM Conference on AI, Ethics, and Society \(AIES 2026\)\. This is an extended version including appendices; the final published version will appear in the AAAI Digital Library\.

![Refer to caption](https://arxiv.org/html/2608.06908v1/x1.png)Figure 1:Pairwise cosine similarity matrices for 2,000 randomly sampled embedding vectors from WikiText\-103 before \(left column\) and after \(right column\) ZCA whitening\.Top:GPT\-2, whose extreme anisotropy \(mean cosine similarity=0\.997=0\.997\) renders the matrix a uniform red field\.Bottom:GloVe \(mean=0\.192=0\.192\), which exhibits moderate anisotropy before whitening\. After calibration, both models converge to near\-zero mean similarity with meaningful pairwise variation restored\.## 1Introduction

Word embeddings\(Mikolovet al\.[2013](https://arxiv.org/html/2608.06908#bib.bib46); Penningtonet al\.[2014](https://arxiv.org/html/2608.06908#bib.bib45)\)encode statistical regularities of language, including social biases that can propagate to downstream applications\(Bolukbasiet al\.[2016](https://arxiv.org/html/2608.06908#bib.bib1); Caliskanet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib2); Garget al\.[2018](https://arxiv.org/html/2608.06908#bib.bib3)\)\. Measuring these biases accurately is necessary for developing equitable NLP systems\. The Word Embedding Association Test\(WEAT; Caliskanet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib2)\)transposes the logic of the Implicit Association Test\(IAT; Greenwaldet al\.[1998](https://arxiv.org/html/2608.06908#bib.bib4)\)from human reaction\-time studies to the geometry of word embedding spaces, measuring differential cosine\-similarity associations between target concepts \(e\.g\., European\-American vs\. African\-American names\) and attribute concepts \(e\.g\., pleasant vs\. unpleasant words\)\. The WEAT has attracted significant attention in both computational social science and AI fairness research, and has been applied in a large number of studies to measure racial, gender, age, and other biases across diverse models and modalities\.

However, the validity of any measurement metric depends on the assumptions under which it operates\. WEAT assumes that cosine similarity faithfully captures semantic associations between words\. This assumption holds when the embedding space is approximately isotropic\(Ethayarajh[2019](https://arxiv.org/html/2608.06908#bib.bib5); Timkey and Van Schijndel[2021](https://arxiv.org/html/2608.06908#bib.bib7); Rudmanet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib65)\)\. In an isotropic space, vectors are distributed with roughly uniform variance in all directions\. Under this condition, the angle between two vectors reflects their semantic relationship rather than a geometric artifact\.

Recent work has shown that this isotropy assumption fails in many language models: static embeddings exhibit dominant mean directions\(Mu and Viswanath[2018](https://arxiv.org/html/2608.06908#bib.bib6)\), contextualized representations occupy narrow cones with near\-unity pairwise similarities\(Ethayarajh[2019](https://arxiv.org/html/2608.06908#bib.bib5)\), and a small number of “rogue dimensions” dominate cosine similarity in Transformers\(Timkey and Van Schijndel[2021](https://arxiv.org/html/2608.06908#bib.bib7)\)\.

Despite this accumulating evidence, the consequences of anisotropy for bias measurement have received surprisingly little direct attention\.Wolfeet al\.\([2024](https://arxiv.org/html/2608.06908#bib.bib8)\)recently introduced ML\-EAT, which provides a comprehensive multilevel evaluation framework and correctly identifies anisotropy as a threat to the validity of cosine\-based bias tests\. However, the purpose of ML\-EAT was to improve the interpretability of EAT measurements, and proposing a correction for anisotropy was outside its scope\. A calibration method that restores the geometric conditions under which cosine\-based bias measurement becomes reliable remains a significant challenge\.

In this paper, we introduce a calibration step that removes anisotropy from language model embedding spaces, restoring the isotropy assumption under which cosine\-based bias measurement is valid\. Specifically, we propose ZCA \(Zero\-phase Component Analysis\) whitening as a pre\-processing calibration for embedding association tests\. ZCA whitening transforms the embedding space to have identity covariance while minimizing the perturbation to the original vectors, making it well suited for effectively reducing anisotropy\. Moreover, whitening has been shown to improve the quality of embeddings on standard semantic similarity benchmarks\(Huanget al\.[2021](https://arxiv.org/html/2608.06908#bib.bib12); Suet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib13)\), suggesting that calibrated embedding spaces may better capture the semantic associations on which bias measurement depends\.

We evaluate our approach across seven embedding models spanning three architectural families \(static, contextualized, and contrastive unsupervised models\) using the ten standard WEAT test suites fromCaliskanet al\.\([2017](https://arxiv.org/html/2608.06908#bib.bib2)\)\. We make the following contributions:

1. 1\.We propose ZCA whitening as a novel pre\-processing calibration method for embedding association tests and show that it substantially reduces anisotropy across all seven models in the embedding space\.
2. 2\.We show that cosine similarity in the calibrated space more faithfully captures semantic associations on standard similarity benchmarks, supporting the validity of bias measurements in the calibrated space\.
3. 3\.We systematically show that calibration changes WEAT outcomes in both directions across 70 model–task combinations, reducing measured associations in some configurations and increasing them in others\.

## 2Related Work

### 2\.1The Embedding Association Test

Caliskanet al\.\([2017](https://arxiv.org/html/2608.06908#bib.bib2)\)introduced the Word Embedding Association Test \(WEAT\), a measurement of intrinsic bias in word embeddings drawing on the design of the IAT\(Greenwaldet al\.[1998](https://arxiv.org/html/2608.06908#bib.bib4)\)\. The WEAT quantifies the relative association of two target groups \(such as European\-American and African\-American names\) with two attribute groups \(such as pleasant and unpleasant words\)\. Given target setsXXandYYand attribute setsAAandBB, the individual association for each target wordwwis:

s​\(w,A,B\)=1\|A\|​∑a∈Acos⁡\(w→,a→\)−1\|B\|​∑b∈Bcos⁡\(w→,b→\)s\(w,A,B\)=\\frac\{1\}\{\|A\|\}\\sum\_\{a\\in A\}\\cos\(\\vec\{w\},\\vec\{a\}\)\-\\frac\{1\}\{\|B\|\}\\sum\_\{b\\in B\}\\cos\(\\vec\{w\},\\vec\{b\}\)\(1\)The test statistic aggregates individual associations across the two target sets:

S​\(X,Y,A,B\)=∑x∈Xs​\(x,A,B\)−∑y∈Ys​\(y,A,B\)S\(X,Y,A,B\)=\\sum\_\{x\\in X\}s\(x,A,B\)\-\\sum\_\{y\\in Y\}s\(y,A,B\)\(2\)The effect sizeddnormalizes this difference:

d=meanx∈X⁡s​\(x,A,B\)−meany∈Y⁡s​\(y,A,B\)stdw∈X∪Y⁡s​\(w,A,B\)d=\\frac\{\\operatorname\{mean\}\_\{x\\in X\}\\,s\(x,A,B\)\-\\operatorname\{mean\}\_\{y\\in Y\}\\,s\(y,A,B\)\}\{\\operatorname\{std\}\_\{w\\in X\\cup Y\}\\,s\(w,A,B\)\}\(3\)This effect size is analogous to Cohen’sdd\(Cohen[1992](https://arxiv.org/html/2608.06908#bib.bib19)\), where values of 0\.2, 0\.5, and 0\.8 are conventionally interpreted as small, medium, and large effects\. Statistical significance is assessed via a one\-sided permutation test that computes thepp\-value as:

p=Pri⁡\[S​\(Xi,Yi,A,B\)\>S​\(X,Y,A,B\)\]p=\\Pr\_\{i\}\\bigl\[S\(X\_\{i\},Y\_\{i\},A,B\)\>S\(X,Y,A,B\)\\bigr\]\(4\)where\{\(Xi,Yi\)\}\\\{\(X\_\{i\},Y\_\{i\}\)\\\}denotes the set of all equally sized partitions ofX∪YX\\cup Y\.

Since its introduction, the WEAT has been extended across modalities and architectures, in order to measure social biases in AI systems\. Text\-based variants include SC\-WEAT for single\-target associations\(Caliskanet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib2)\), SEAT for sentence\-level embeddings\(Mayet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib15)\), a pretraining\-objective\-based adaptation\(Kuritaet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib14)\), and CEAT, which treats contextualization as a random effect at the word level\(Guo and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib16)\)\. The framework has also been extended beyond text to image encoders \(iEAT;Steed and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib17)\), grounded vision\-and\-language models \(Grounded\-WEAT;Rosset al\.[2021](https://arxiv.org/html/2608.06908#bib.bib20)\), CLIP\(Wolfe and Caliskan[2022](https://arxiv.org/html/2608.06908#bib.bib21); Wolfeet al\.[2022a](https://arxiv.org/html/2608.06908#bib.bib22); Radfordet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib23)\), speech models \(SpEAT;Slaughteret al\.[2023](https://arxiv.org/html/2608.06908#bib.bib18)\), and text\-to\-video generators \(VEAT;Sunet al\.[2026](https://arxiv.org/html/2608.06908#bib.bib66)\)\. Recently,Wolfeet al\.\([2024](https://arxiv.org/html/2608.06908#bib.bib8)\)proposed ML\-EAT, which provides multilevel interpretable bias measurements and includes an anisotropy diagnostic\. All of these variants rely on cosine similarity and are therefore subject to the same geometric artifacts that this paper addresses\. Their widespread adoption across NLP, computer vision, and speech processing underscores the importance of ensuring that their measurements are geometrically sound\.

### 2\.2Applications of EATs in Social Science

Embedding association tests have been widely adopted in the social sciences as a tool for studying societal phenomena at scale\. A growing body of work uses EATs to probe contemporary cultural patterns in large corpora, including gender defaults and stereotypes in internet English\(Caliskanet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib24); Baileyet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib25)\), the relationship between gender bias and societal structure\(Napp[2023](https://arxiv.org/html/2608.06908#bib.bib26)\), and temporal shifts in Wikipedia\(Schmahlet al\.[2020](https://arxiv.org/html/2608.06908#bib.bib27)\)\. Cross\-cultural and cross\-linguistic studies have extended this reach to dozens of languages\(Lewis and Lupyan[2020](https://arxiv.org/html/2608.06908#bib.bib28); Charlesworthet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib29); Mukherjeeet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib30)\)\. Historical corpora have enabled the study of how these associations evolve over time, including a century of gender and ethnic stereotypes\(Garget al\.[2018](https://arxiv.org/html/2608.06908#bib.bib3)\), 200 years of social\-group stereotypes\(Charlesworthet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib31)\), intersectional stereotypes\(Charlesworthet al\.[2024b](https://arxiv.org/html/2608.06908#bib.bib32); Borensteinet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib33)\), xenophobia in 19th–20th century travel literature\(Sunsay[2023](https://arxiv.org/html/2608.06908#bib.bib34)\), and the expansion of the societal “moral circle”\(Leachet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib35)\)\. Collectively, these applications demonstrate that EATs can serve not only as a measure of contemporary associations but also as a tool for tracing long\-term societal change\.

In applied domains, EATs have been used to measure biases in biomedical research\(Rioset al\.[2020](https://arxiv.org/html/2608.06908#bib.bib36)\), clinical notes\(Cobertet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib71)\), legal opinions\(Matthewset al\.[2022](https://arxiv.org/html/2608.06908#bib.bib37)\), and court proceedings\(Duttaet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib38)\)\.Gray and Wu \([2025](https://arxiv.org/html/2608.06908#bib.bib74)\)further proposed SD\-WEAT, a variant handling multi\-level attribute groups for healthcare bias benchmarking\. Recent work has also linked embedding\-based measurements to human\-level implicit associations, demonstrating meaningful correspondence between embedding biases and societal stereotypes\(Charlesworthet al\.[2024a](https://arxiv.org/html/2608.06908#bib.bib73); Morehouseet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib72); Manziniet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib75)\)\.

These diverse applications underscore that the validity of EAT measurements is not a narrow technical concern but a matter of scientific integrity across multiple disciplines\. However, several studies have identified potential problems with methods that rely on cosine similarity between embedding vectors, suggesting that the geometric properties of the embedding space may affect the reliability of such measurements\.

### 2\.3Embedding Anisotropy and Its Implications for WEAT

Anisotropy in word embeddings describes a geometric property in which vectors cluster in a narrow region of the embedding space\(Caiet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib43)\), causing cosine similarities between arbitrary word pairs to be systematically high\.Ethayarajh \([2019](https://arxiv.org/html/2608.06908#bib.bib5)\)demonstrated that contextualized representations from BERT, ELMo, and GPT\-2 occupy a narrow cone with average cosine similarities approaching 1\.0 in upper layers, whileMu and Viswanath \([2018](https://arxiv.org/html/2608.06908#bib.bib6)\)identified dominant principal components in word2vec and GloVe, andTimkey and Van Schijndel \([2021](https://arxiv.org/html/2608.06908#bib.bib7)\)showed that a few “rogue dimensions” dominate cosine computation in Transformers\. Further work has documented additional effects\(Zhouet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib39); Godeyet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib40); Machina and Mercer[2024](https://arxiv.org/html/2608.06908#bib.bib41); Rajaee and Pilehvar[2022](https://arxiv.org/html/2608.06908#bib.bib42)\)\.

For WEAT, such geometric distortion can compress or inflate the differential cosine associations that define its effect size, producing measurements that do not accurately reflect the underlying bias structure\.Wolfeet al\.\([2024](https://arxiv.org/html/2608.06908#bib.bib8)\)incorporated an anisotropy diagnostic into ML\-EAT that flags unreliable models, but ML\-EAT aims at interpretability rather than correction\.Mu and Viswanath \([2018](https://arxiv.org/html/2608.06908#bib.bib6)\)proposed removing top principal components, but this requires explicitly discarding information and selecting a model\-dependent hyperparameter, limiting its practical applicability\. Thus, a calibration method that restores the reliability of cosine\-based measurement without requiring such choices is desirable\.

## 3Approach

### 3\.1How Anisotropy Distorts Cosine\-Based Association Tests

Cosine similarity between two vectorsu→,v→∈ℝd\\vec\{u\},\\vec\{v\}\\in\\mathbb\{R\}^\{d\}is defined as:

cos⁡\(u→,v→\)=u→⋅v→‖u→‖​‖v→‖\\cos\(\\vec\{u\},\\vec\{v\}\)=\\frac\{\\vec\{u\}\\cdot\\vec\{v\}\}\{\\\|\\vec\{u\}\\\|\\,\\\|\\vec\{v\}\\\|\}\(5\)For cosine similarity to serve as a reliable measure of semantic association, the embedding space should be approximately isotropic\(Ethayarajh[2019](https://arxiv.org/html/2608.06908#bib.bib5); Wolfeet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib8)\)\. Specifically, the covariance matrixΣ\\Sigmaof the embedding vectors should be approximately proportional to the identity matrix\(Rudmanet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib65)\):

Σ≈σ2​I\\Sigma\\approx\\sigma^\{2\}I\(6\)
WhenΣ\\Sigmadeparts from proportionality toII, cosine similarity may become systematically distorted\. Because cosine similarity is dominated by high\-variance directions, vectors can appear more similar than they semantically are, while meaningful differences along low\-variance directions are masked\(Mu and Viswanath[2018](https://arxiv.org/html/2608.06908#bib.bib6); Timkey and Van Schijndel[2021](https://arxiv.org/html/2608.06908#bib.bib7)\)\.

Since the WEAT computes differential cosine\-similarity associations between target and attribute word sets \(Equations[1](https://arxiv.org/html/2608.06908#S2.E1)–[3](https://arxiv.org/html/2608.06908#S2.E3)\), the effect sizeddmay be affected by anisotropy if the distortion is not uniform across the word sets involved in the test\. This suggests that anisotropy could both inflate and compress genuine bias signals, depending on the relationship between the word sets and the covariance structure of the embedding space\.

To address this potential distortion, a calibration step that transforms the covariance matrix of the embedding space toward the identity matrix is needed, thereby restoring the geometric conditions under which cosine similarity can function as a reliable measure of semantic association\.

### 3\.2ZCA Whitening

We adopt Zero\-phase Component Analysis \(ZCA\) whitening as the pre\-processing calibration step\.

Given a set of embedding vectors\{x1,x2,…,xn\}⊂ℝd\\\{x\_\{1\},x\_\{2\},\\ldots,x\_\{n\}\\\}\\subset\\mathbb\{R\}^\{d\}, we first compute the sample meanμ\\muand the sample covariance matrixΣ\\Sigma:

μ=1n​∑i=1nxi,Σ=1n−1​∑i=1n\(xi−μ\)​\(xi−μ\)⊤\\mu=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}x\_\{i\},\\quad\\Sigma=\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\mu\)\(x\_\{i\}\-\\mu\)^\{\\top\}\(7\)We then center each vector by subtracting the mean and apply the ZCA whitening matrixWZCAW\_\{\\text\{ZCA\}\}to obtain the transformed vectors:

zi=WZCA​\(xi−μ\),WZCA=Σ−1/2z\_\{i\}=W\_\{\\text\{ZCA\}\}\(x\_\{i\}\-\\mu\),\\quad W\_\{\\text\{ZCA\}\}=\\Sigma^\{\-1/2\}\(8\)whereΣ−1/2\\Sigma^\{\-1/2\}denotes the symmetric matrix square root ofΣ−1\\Sigma^\{\-1\}\. To compute this matrix, we perform the eigendecomposition of the covariance matrixΣ=U​Λ​U⊤\\Sigma=U\\Lambda U^\{\\top\}, whereUUis an orthogonal matrix whose columns are the eigenvectors andΛ\\Lambdais a diagonal matrix of the corresponding eigenvalues\. The whitening matrix then takes the explicit form:

WZCA=U​Λ−1/2​U⊤W\_\{\\text\{ZCA\}\}=U\\Lambda^\{\-1/2\}U^\{\\top\}\(9\)This transformation rescales each eigenvector direction by the inverse square root of its eigenvalue: dimensions with disproportionately large variance are compressed, while dimensions with small variance are expanded, resulting in uniform variance across all directions\.

The transformed vectors\{zi\}\\\{z\_\{i\}\\\}have identity covariance by construction:

Cov⁡\(z\)=WZCA​Σ​WZCA⊤=Σ−1/2​Σ​Σ−1/2=I,\\operatorname\{Cov\}\(z\)=W\_\{\\text\{ZCA\}\}\\,\\Sigma\\,W\_\{\\text\{ZCA\}\}^\{\\top\}=\\Sigma^\{\-1/2\}\\,\\Sigma\\,\\Sigma^\{\-1/2\}=I,\(10\)where the final equality follows from the symmetry ofWZCA=Σ−1/2W\_\{\\text\{ZCA\}\}=\\Sigma^\{\-1/2\}\(see Appendix A for a full derivation\)\. SinceCov⁡\(z\)=I\\operatorname\{Cov\}\(z\)=I, the calibrated embedding space satisfies the isotropy condition \(Equation[6](https://arxiv.org/html/2608.06908#S3.E6)withσ2=1\\sigma^\{2\}=1\) under which cosine similarity can function as a reliable measure of semantic association\.

Among the family of whitening transformations that produce identity covariance, ZCA uniquely minimizes the expected squared distance between original and transformed vectors\(Kessyet al\.[2018](https://arxiv.org/html/2608.06908#bib.bib11)\), making it particularly suitable for bias measurement calibration where preserving the original semantic structure is essential\. Moreover, whitening has been shown to substantially improve scores on standard semantic textual similarity benchmarks\(Huanget al\.[2021](https://arxiv.org/html/2608.06908#bib.bib12); Suet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib13)\), suggesting that the calibrated embedding space more faithfully captures semantic associations between words and may therefore provide a more reliable basis for extracting social associations that were obscured by anisotropy in the original space\.

### 3\.3Calibration Pipeline

Our full measurement pipeline operates in three stages\.

In the first stage, we estimate the whitening matrix from a large reference sample of embedding vectors\. We adopt WikiText\-103\(Merityet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib51)\)as the reference corpus for collecting these samples\. From this corpus, we draw 100,000 samples and feed them into the model under evaluation to obtain the corresponding embedding vectors\. Using the resulting embeddings, we compute the sample meanμ\\muand the covariance matrixΣ\\Sigma\(Equation[7](https://arxiv.org/html/2608.06908#S3.E7)\), from which we estimate the ZCA whitening matrixWZCAW\_\{\\text\{ZCA\}\}\(Equation[9](https://arxiv.org/html/2608.06908#S3.E9)\)\. This estimation is performed once per model and the resulting whitening statistics are reused across all test suites\.

In the second stage, we apply the estimated whitening transformation to the specific word vectors involved in the WEAT test suites\. Each word vector is centered by subtracting the reference meanμ\\muand then transformed byWZCAW\_\{\\text\{ZCA\}\}to obtain the calibrated vector \(Equation[8](https://arxiv.org/html/2608.06908#S3.E8)\)\.

In the third stage, we compute WEAT scores \(effect sizeddand permutationpp\-value\) using the calibrated vectors\.

This pipeline is modular: it can be applied to any embedding model and any cosine\-similarity\-based bias metric without modifying the metric itself\.

## 4Experimental Setup

Table 1:The ten standard WEAT test suites\(Caliskanet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib2)\)\. Each suite specifies two target sets \(XX,YY\) and two attribute sets \(AA,BB\)\.\|⋅\|\|\{\\cdot\}\|denotes the number of stimulus words in each set, following the lists provided inCaliskanet al\.\([2017](https://arxiv.org/html/2608.06908#bib.bib2)\)\. EA and AA denote European\-American and African\-American, respectively\.### 4\.1Models

We evaluate seven embedding models spanning three architectural families\.

##### Static embeddings\.

GloVe\(Penningtonet al\.[2014](https://arxiv.org/html/2608.06908#bib.bib45)\)trained on Wikipedia and Gigaword \(6B tokens, 100 dimensions\), and word2vec\(Mikolovet al\.[2013](https://arxiv.org/html/2608.06908#bib.bib46)\)trained on Google News \(approximately 100B tokens, 300 dimensions\)\. We use the 6B\-token, 100\-dimensional GloVe model rather than the 840B\-token, 300\-dimensional variant used inCaliskanet al\.\([2017](https://arxiv.org/html/2608.06908#bib.bib2)\)\. Our purpose is not to replicate their specific results but to evaluate the impact of anisotropy calibration, and the 6B model exhibits sufficient anisotropy \(Table[2](https://arxiv.org/html/2608.06908#S5.T2)\) to serve this purpose\.

##### Contextualized embeddings\.

BERT\-base\-cased\(Devlinet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib47)\), RoBERTa\-base\(Liuet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib48)\), and GPT\-2\(Radfordet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib49)\), each with 12 Transformer layers and 768\-dimensional hidden representations\. For all Transformer models, we use representations from the final layer, consistent with prior work applying EATs to contextualized models\(Mayet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib15); Guo and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib16); Wolfeet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib8)\)\. Word\-level embeddings are obtained following the Aggregated procedure ofBommasaniet al\.\([2020](https://arxiv.org/html/2608.06908#bib.bib53)\): for each target or attribute word, we sample up to 20 sentences containing that word from a preprocessed WikiText\-103 corpus\. Preprocessing retains sentences of 7–75 tokens and excludes section headers, yielding up to 200,000 indexed sentences with a fixed random seed for reproducibility\. Within each sentence, subword tokens of the target word are mean\-pooled into a single contextualized vector, and the resulting vectors across the 20 sentences are then averaged to produce one word\-level representation\.Bommasaniet al\.\([2020](https://arxiv.org/html/2608.06908#bib.bib53)\)reported that bias evaluation results were fairly stable acrossni∈\{20,50,100\}n\_\{i\}\\in\\\{20,50,100\\\}contexts per word\. We therefore adoptedni=20n\_\{i\}=20to maintain consistency with their protocol and to minimize computational overhead\. We applied the canonical WEAT to these word\-level representations rather than sentence\-level variants such as SEAT\(Mayet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib15)\)or distributional variants such as CEAT\(Guo and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib16)\)\. This choice allows us to use a unified measurement procedure across all seven models, including static embeddings for which SEAT and CEAT are not applicable\.

##### Unsupervised contrastive embeddings\.

We adopt two unsupervised contrastive models from the SimCSE framework\(Gaoet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib50)\): unsup\-SimCSE\-BERT\-base\-uncased \(unsup\-BERT\) and unsup\-SimCSE\-RoBERTa\-base \(unsup\-RoBERTa\)\. Both models share the same Transformer architecture as their supervised counterparts \(12 layers, 768\-dimensional hidden representations\), and word\-level embeddings are obtained using the same aggregation procedure ofBommasaniet al\.\([2020](https://arxiv.org/html/2608.06908#bib.bib53)\)described above\. These models differ from the standard contextualized models in two key aspects: training signal \(unsupervised vs\. supervised\) and objective function \(contrastive vs\. language modeling\)\. We include them in our analysis to examine how these factors influence the degree of anisotropy and the patterns of bias detected by WEAT\.

##### Whitening statistics\.

The whitening matrixWZCAW\_\{\\text\{ZCA\}\}is estimated from WikiText\-103\(Merityet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib51)\)\. For static models, we sample up to 100,000 vocabulary items\. For contextualized and unsupervised contrastive models, we sample up to 100,000 sentences\. These contextualized models are pretrained on sentence\-level inputs\. Feeding isolated words may therefore produce out\-of\-distribution representations\(Bommasaniet al\.[2020](https://arxiv.org/html/2608.06908#bib.bib53)\)\. To avoid this issue, we use sentence\-level inputs to estimate the covariance matrix\. To ensure numerical stability when computing the whitening matrix \(Equation[9](https://arxiv.org/html/2608.06908#S3.E9)\), we regularize the eigenvalues by replacingΛ\\LambdawithΛ\+ε​I\\Lambda\+\\varepsilon Ibefore inversion, yielding:

WZCA=U​\(Λ\+ε​I\)−1/2​U⊤W\_\{\\text\{ZCA\}\}=U\(\\Lambda\+\\varepsilon I\)^\{\-1/2\}U^\{\\top\}\(11\)whereε=10−3\\varepsilon=10^\{\-3\}\. This prevents near\-zero eigenvalues from producing excessively large entries in the whitening matrix\.

### 4\.2WEAT Test Suites

We use the ten standard WEAT test suites fromCaliskanet al\.\([2017](https://arxiv.org/html/2608.06908#bib.bib2)\)\(Table[1](https://arxiv.org/html/2608.06908#S4.T1)\)\. These test suites correspond to well\-established IAT findings in social psychology and have been used in numerous studies to benchmark bias in NLP models\(Kuritaet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib14); Mayet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib15); Guo and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib16); Wolfeet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib8)\)\.

![Refer to caption](https://arxiv.org/html/2608.06908v1/x2.png)Figure 2:Explained variance ratio of the top 50 principal components \(log scale\) before \(left\) and after \(right\) ZCA whitening for all seven models\.Left:Contextualized models \(GPT\-2, RoBERTa, BERT\) show extreme variance concentration in the first few components, while static models \(GloVe, word2vec\) and contrastive models \(unsup\-BERT, unsup\-RoBERTa\) exhibit flatter distributions\.Right:After whitening, all models converge to a near\-uniform eigenvalue distribution\.
### 4\.3Evaluation Protocol

##### Experiment 1: Anisotropy measurement\.

We quantify anisotropy by computing the mean and standard deviation of pairwise cosine similarities among 2,000 randomly sampled embedding vectors from WikiText\-103, both before and after ZCA whitening\. An isotropic space yields a mean pairwise cosine similarity near zero\. In addition, we qualitatively compare the embedding spaces before and after whitening by examining the explained variance ratio of the covariance matrices, which illustrates how variance is distributed across dimensions\. We also visualize pairwise cosine similarity matrices over the sampled vectors\.

##### Experiment 2: Impact of anisotropy on semantic structure\.

We evaluate how anisotropy affects the quality of cosine\-similarity\-based semantic measurements by computing Spearman rank correlations \(ρ\\rho\) on WordSim\-353\(Finkelsteinet al\.[2002](https://arxiv.org/html/2608.06908#bib.bib54)\), SimLex\-999\(Hillet al\.[2015](https://arxiv.org/html/2608.06908#bib.bib55)\), and STS\-B\(Ceret al\.[2017](https://arxiv.org/html/2608.06908#bib.bib76)\)before and after whitening\. Each benchmark provides word or sentence pairs with human\-assigned similarity scores\. A higher Spearman correlation indicates that cosine similarity better captures the semantic associations in these benchmarks\. Statistical significance is assessed via paired bootstrap resampling \(10,000 iterations\)\. If calibration improves these correlations, it suggests that cosine similarity in the calibrated space more faithfully captures semantic associations, supporting the validity of bias measurements conducted in that space\.

##### Experiment 3: Bias measurement with WEAT\.

For each model–task combination, we compute the WEAT effect sizeddand permutationpp\-value \(100,000 permutations\) on both raw and whitened embeddings\. Results are classified based on the change in statistical significance atp<0\.05p<0\.05:Stable, significance unchanged;Disappearing, significant before whitening but non\-significant after;Emerging, non\-significant before but significant after whitening\. We then analyze, for each bias category, how these classifications distribute across models and identify the systematic patterns of distortion induced by anisotropy\.

## 5Results

Table 2:Anisotropy measured by mean pairwise cosine similarity \(±\\pmstandard deviation, SD\) among 2,000 randomly sampled vectors before \(Raw\) and after \(White\) ZCA whitening\.### 5\.1ZCA Whitening Reduces Anisotropy

Table[2](https://arxiv.org/html/2608.06908#S5.T2)presents the mean pairwise cosine similarity before and after ZCA whitening for all seven models\. Static embeddings exhibit mild anisotropy \(GloVe: 0\.192; word2vec: 0\.059\), while contextualized models display severe anisotropy, with GPT\-2 reaching an extreme value of 0\.997 \(SD = 0\.003\) in which virtually all vectors point in nearly the same direction\. This pattern is consistent with prior reports of cosine similarities concentrating near 1\.0 between arbitrary word pairs in GPT\-2 Base\(Timkey and Van Schijndel[2021](https://arxiv.org/html/2608.06908#bib.bib7); Wolfeet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib8)\)\. The unsupervised contrastive models show intermediate anisotropy \(unsup\-BERT: 0\.386; unsup\-RoBERTa: 0\.475\), consistent with the uniformity\-promoting nature of contrastive learning\(Gaoet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib50)\)but still substantial enough to distort cosine\-based measurements\. After ZCA whitening, all models converge to near\-zero mean pairwise cosine similarity \(0\.001–0\.040\), confirming that calibration reduces anisotropy regardless of the initial degree of distortion\. This reduction is also visible in the pairwise cosine similarity matrices \(Figure[1](https://arxiv.org/html/2608.06908#S0.F1)\)\. GPT\-2’s matrix appears as a nearly uniform red field before whitening, while GloVe exhibits moderate anisotropy\. After calibration, both center near zero with restored pairwise variation\. The same qualitative pattern holds for the other five models \(see Appendix B\)\.

Figure[2](https://arxiv.org/html/2608.06908#S4.F2)provides further confirmation through the explained variance ratio of each model’s embedding covariance matrix\. Before whitening, the contextualized models show extreme variance concentration in the first few principal components, whereas the static and contrastive models exhibit flatter but still non\-uniform distributions\. After whitening, the explained variance ratio flattens substantially across all models, indicating that dominant directional components have been removed and variance is distributed uniformly across dimensions\.

### 5\.2Calibrated Embeddings Better Capture Semantic Associations

Table[3](https://arxiv.org/html/2608.06908#S5.T3)reports Spearman rank correlations on three semantic similarity benchmarks before and after ZCA whitening\. For models with high anisotropy \(GPT\-2, RoBERTa, and BERT\), whitening produces substantial improvements on word\-level benchmarks \(WordSim\-353 and SimLex\-999\)\. The most dramatic gains appear for GPT\-2, where Spearmanρ\\rhoon WordSim\-353 increases from 0\.263 to 0\.620 \(Δ=\+0\.358\\Delta=\+0\.358,p<0\.001p<0\.001\) and on SimLex\-999 from 0\.097 to 0\.411 \(Δ=\+0\.314\\Delta=\+0\.314,p<0\.001p<0\.001\), suggesting that the original embedding space was so anisotropic that cosine similarity was largely non\-functional and whitening restored much of its discriminative capacity\. This pattern is consistent withTimkey and Van Schijndel \([2021](https://arxiv.org/html/2608.06908#bib.bib7)\), who showed that correcting for rogue dimensions substantially improves cosine similarity as a measure of semantic association\. RoBERTa and BERT show similar word\-level gains \(Δ=\+0\.054\\Delta=\+0\.054to\+0\.191\+0\.191\), while their STS\-B scores are either unchanged \(RoBERTa\) or modestly decreased \(BERT:Δ=−0\.055\\Delta=\-0\.055\)\.

For word2vec, which has the lowest initial anisotropy \(mean cosine 0\.059\), the results are mixed and small in magnitude \(\|Δ\|≤0\.027\|\\Delta\|\\leq 0\.027\), suggesting that whitening neither substantially improves nor degrades an already near\-isotropic space\. The unsupervised contrastive models show essentially unchanged word\-level scores but small, significant decreases on STS\-B \(Δ=−0\.055\\Delta=\-0\.055for unsup\-BERT;Δ=−0\.034\\Delta=\-0\.034for unsup\-RoBERTa\), possibly reflecting a domain mismatch between the reference corpus \(WikiText\-103\) and the data on which these models were originally optimized\.

Taken together, these results indicate that ZCA whitening preserves the semantic structure of the embedding space\. In many cases, it also enables cosine similarity to more faithfully capture semantic associations\. This improvement is particularly pronounced for highly anisotropic models, where calibration is most needed\. The modest STS\-B decreases warrant acknowledgment but are small relative to the word\-level gains\. Because WEAT measures bias through cosine similarity, these findings support the interpretation that WEAT measurements in the calibrated space are more reliable indicators of genuine semantic associations, particularly for models with high initial anisotropy\.

Table 3:Spearmanρ\\rhoon semantic similarity benchmarks before \(Raw\) and after \(White\) ZCA whitening\.Δ\\Deltadenotes the change\. Significance:p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001,p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗<0\.05\{\}^\{\*\}p<0\.05, n\.s\. = not significant \(paired bootstrap, 10k iterations\)\.
### 5\.3Anisotropy Distorts WEAT in Both Directions

![Refer to caption](https://arxiv.org/html/2608.06908v1/x3.png)Figure 3:WEAT effect sizes \(dd\) before \(upper\-left triangle\) and after \(lower\-right triangle\) ZCA whitening across seven models and ten test suites\. Color intensity encodes effect size magnitude \(clipped to\[−1\.5,1\.5\]\[\-1\.5,1\.5\]\)\. Bold values with asterisks and bordered cells indicate statistical significance \(p<0\.05p<0\.05\); regular\-weight values without borders indicate non\-significant results\. Models are grouped by architectural family: static \(top\), contextualized \(middle\), and unsupervised contrastive \(bottom\)\. Test suites are grouped by bias type: Valence \(W1–W2\), Race \(W3–W5\), Gender \(W6–W8\), Health \(W9\), and Age \(W10\)\.Figure[3](https://arxiv.org/html/2608.06908#S5.F3)visualizes WEAT effect sizes before and after ZCA whitening across all 70 model–task combinations, and Figure[4](https://arxiv.org/html/2608.06908#S5.F4)plots whitened versus raw effect sizes grouped by bias type\. Full numerical results, including effect sizes,pp\-values, and significance change classifications for all combinations, are provided in Appendix C\. Across the 70 combinations, we observe 48 Stable cases \(68\.6%\), 12 Disappearing cases \(17\.1%\), and 10 Emerging cases \(14\.3%\)\. Effect size shifts also occur within Stable cases\. Among the Stable cases that remain significant in both spaces, 7 exhibit\|Δ​d\|\>0\.5\|\\Delta d\|\>0\.5, including GPT\-2 on W3 \(d=1\.24→0\.57d=1\.24\\to 0\.57\) and unsup\-RoBERTa on W6 \(d=1\.61→0\.99d=1\.61\\to 0\.99\)\. In total, 29 of 70 combinations \(41\.4%\) show either a significance change or a substantial effect\-size shift \(\|Δ​d\|\>0\.5\|\\Delta d\|\>0\.5\) after calibration, suggesting that anisotropy is a practical source of measurement error\.

#### Valence \(W1–W2\)\.

Most valence measurements cluster near or above the diagonal in Figure[4](https://arxiv.org/html/2608.06908#S5.F4)\(a\), indicating that effect sizes are preserved or increase after whitening\. Static models retain significance throughout, though word2vec shows compression of inflated effects \(W2:d=1\.63→1\.03d=1\.63\\to 1\.03\)\. The contextualized models show a markedly different pattern: four of six contextualized model–task combinations exhibit Emerging bias \(BERT W2, RoBERTa W2, GPT\-2 W1, GPT\-2 W2\)\. The most striking case is GPT\-2 on W2 \(d=−0\.87→d=1\.05d=\-0\.87\\to d=1\.05\)\. BERT and RoBERTa on W2, and GPT\-2 on W1, follow the same pattern \(see Appendix C\)\. Because valence associations such as flowers–pleasant and weapons–unpleasant are among the most robust findings in the IAT literature\(Greenwaldet al\.[1998](https://arxiv.org/html/2608.06908#bib.bib4)\), their absence in uncalibrated contextualized embeddings likely reflects geometric distortion rather than a genuine lack of association\. The contrastive models are mostly Stable, with one Disappearing case \(unsup\-RoBERTa W2:d=1\.47→0\.15d=1\.47\\to 0\.15\)\.

#### Race \(W3–W5\)\.

Race measurements are predominantly located below the diagonal in Figure[4](https://arxiv.org/html/2608.06908#S5.F4)\(b\), indicating systematic decreases in effect size after whitening\. This category concentrates the bulk of overestimation: 8 of 12 Disappearing cases \(66\.7%\) occur in W3–W5\. Both static models show Disappearing bias on W3 \(e\.g\., GloVe:d=0\.95→0\.24d=0\.95\\to 0\.24\), with reduced but still significant effects on W4 and W5\. Among the contextualized models, BERT exhibits Disappearing bias across all three race tests, and GPT\-2 on W4 and W5 \(full transitions in Appendix C\); RoBERTa is the exception, maintaining stable significance throughout\. The contrastive models show only one Disappearing case within the race tests \(unsup\-BERT W3\), and unsup\-RoBERTa even shows slightly increased race effects after whitening \(W4:d=0\.86→1\.07d=0\.86\\to 1\.07\)\.

#### Gender \(W6–W8\)\.

Gender measurements show considerable scatter around the diagonal in Figure[4](https://arxiv.org/html/2608.06908#S5.F4)\(c\), reflecting coexistence of overestimated and underestimated bias\. Static models retain strong significance on W6 \(word2vec:d=1\.89→1\.81d=1\.89\\to 1\.81; GloVe:d=1\.72→1\.63d=1\.72\\to 1\.63\), but word2vec shows Disappearing cases on W7 and W8\. The contextualized models reveal a more nuanced pattern: only one of nine combinations shows Emerging bias \(BERT on W8:d=0\.48→1\.15d=0\.48\\to 1\.15\)\. GPT\-2 shows directional reversals on W7 \(d=−0\.58→0\.69d=\-0\.58\\to 0\.69\) and W8 \(d=−0\.54→0\.81d=\-0\.54\\to 0\.81\), where raw associations flip from negative to positive after whitening\. This suggests that severe anisotropy can invert the apparent sign of association\. The contrastive models provide further evidence: unsup\-BERT shows Emerging bias on both W7 \(d=0\.47→1\.18d=0\.47\\to 1\.18\) and W8 \(d=0\.59→0\.98d=0\.59\\to 0\.98\)\. These emerging effect sizes are comparable to or larger than those in the standard contextualized models, suggesting that while contrastive training reduces geometric distortion, it does not eliminate the bias encoded in the pre\-training data\.

#### Health and Age \(W9–W10\)\.

For health \(W9\), the most notable result is GPT\-2’s Emerging case \(d=−0\.01→d=1\.55d=\-0\.01\\to d=1\.55\), where whitening uncovers a strong association that was geometrically masked in the raw space\. Unsup\-BERT also shows an Emerging case and unsup\-RoBERTa a Disappearing one on W9, while static models show mixed patterns \(see Appendix C\)\. For age \(W10\), six of seven models show Stable results, with one Emerging case \(unsup\-RoBERTa:d=0\.49→d=0\.92d=0\.49\\to d=0\.92\); the contextualized models maintain large significant effects \(d\>1\.0d\>1\.0after whitening\)\.

![Refer to caption](https://arxiv.org/html/2608.06908v1/x4.png)Figure 4:Scatter plots of WEAT effect sizes before \(x\-axis\) and after \(y\-axis\) ZCA whitening, separated by bias type: \(a\) Valence, \(b\) Race, \(c\) Gender, and \(d\) Other \(Health & Age\)\. The diagonal line represents no change; points above the line indicate increased effect sizes after whitening \(underestimation in the raw space\), and points below indicate decreased effect sizes \(overestimation\)\. Marker color denotes significance change: Disappearing \(red\), Stable \(pink\), and Emerging \(blue\)\. Marker shape denotes architectural family: circle \(static\), triangle \(contextualized\), and square \(contrastive\)\.

### 5\.4The Role of Anisotropy Severity

The frequency of significance changes appears broadly consistent with the degree of initial anisotropy: GPT\-2 \(mean cosine 0\.997\) shows changes in 5 of 10 tests, BERT \(0\.821\) in 5, unsup\-BERT \(0\.386\) in 4, while word2vec \(0\.059\) shows changes in 3 of 10 tests\. However, RoBERTa \(0\.954\) shows only 1 change despite its high anisotropy, and GloVe \(0\.192\) likewise shows only 1 change, indicating that the relationship between anisotropy severity and the frequency of significance changes is not strictly monotonic and may depend on additional factors such as the specific geometry of the embedding space relative to the WEAT stimulus words\.

### 5\.5Patterns of Distortion Across Bias Categories and Embedding Families

The category\-level analysis reveals two systematic asymmetries, summarized visually in Figure[4](https://arxiv.org/html/2608.06908#S5.F4)\. First, distortion direction depends on bias category: Disappearing cases concentrate in the race tests, which account for 8 of 12 \(66\.7%\), while Emerging cases are most frequent in the valence tests \(4 of 10\), with the remainder split evenly between gender \(3\) and health/age \(3\)\. This asymmetry suggests that anisotropy may interact differently with different stimulus configurations\. However, the specific mechanism behind this pattern is not fully clear from our experimental results alone\. One possibility is that the geometric relationship between target sets and the dominant variance directions of the embedding space differs across bias categories\. As a result, anisotropy may amplify cosine differences in some cases and compress them in others\. Identifying the precise factors that determine the direction of distortion is an important direction for future work\.

Second, distortion direction depends on architectural family\. The contextualized models, which exhibit the highest anisotropy \(mean cosine 0\.821–0\.997\), account for 6 of 10 Emerging cases \(60\.0%\), consistent with the interpretation that severe anisotropy compresses the cosine similarity range until genuine differential associations fall below the significance threshold\. The static models, despite their relatively low anisotropy, contribute 4 of 12 Disappearing cases \(33\.3%\), indicating that even moderate distortion can inflate specific measurements\. The contrastive models exhibit both directions of distortion across categories, indicating that contrastive training reduces geometric distortion but neither prevents it uniformly nor eliminates the bias encoded in the pre\-training data\.

## 6Discussion

### 6\.1Implications for Computational Social Science

Our findings have direct implications for intrinsic bias measurement\. The observation that approximately 30% of WEAT measurements change significance status after geometric calibration suggests that a substantial fraction of assessments based on uncalibrated embeddings may be unreliable\. Practitioners conducting bias audits should therefore perform an isotropy check before applying cosine\-based measurements and calibrate when anisotropy is detected\.

The WEAT has been widely used in computational social science to study societal biases that are difficult or costly to measure experimentally, including historical shifts in stereotypes over 200 years\(Charlesworthet al\.[2022](https://arxiv.org/html/2608.06908#bib.bib31)\), 100 years of gender and ethnic stereotypes\(Garget al\.[2018](https://arxiv.org/html/2608.06908#bib.bib3)\), gender stereotypes across 25 languages\(Lewis and Lupyan[2020](https://arxiv.org/html/2608.06908#bib.bib28)\), and intersectional stereotypes\(Charlesworthet al\.[2024b](https://arxiv.org/html/2608.06908#bib.bib32)\)\. Our results suggest that conclusions drawn from highly anisotropic models may need to be revisited with calibrated measurements to determine whether reported biases were overestimated, underestimated, or accurately captured\. This does not invalidate prior findings; rather, geometric calibration offers a tool to assess their robustness\.

The bidirectional nature of the distortion makes this concern particularly pressing\. If anisotropy only inflated bias measurements, uncalibrated results could be interpreted as upper bounds\. However, because anisotropy can also mask genuine biases, uncalibrated results cannot be treated as conservative estimates either\.

Our work parallels methodological debates in psychology about how implicit bias should be measured\. Just asGreenwaldet al\.\([2022](https://arxiv.org/html/2608.06908#bib.bib77)\)provided recommended best practices for research using the IAT, we argue that computational bias measures require analogous methodological discipline, including correction for known geometric distortions\.Morehouseet al\.\([2025](https://arxiv.org/html/2608.06908#bib.bib58)\)likewise called for importing social science best practices into LLM bias probing\. Our ZCA calibration addresses one specific aspect of measurement invariance: ensuring that WEAT scores are comparable across models with differing degrees of anisotropy\.

The importance of calibrating intrinsic bias measurements is reinforced by growing evidence that such biases propagate to downstream behavior\. EAT\-measured biases in vision\-language models correlate with downstream task performance and propagate to zero\-shot retrieval\(Ghateet al\.[2025](https://arxiv.org/html/2608.06908#bib.bib62)\), image classification\(Wolfeet al\.[2022b](https://arxiv.org/html/2608.06908#bib.bib68)\), visual question answering, captioning, and generation\(Wolfe and Caliskan[2022](https://arxiv.org/html/2608.06908#bib.bib21); Friedrichet al\.[2025](https://arxiv.org/html/2608.06908#bib.bib69)\), and sentiment classification\(Meiet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib67)\)\. Biased AI outputs have also been shown to shape humans’ own implicit associations\(Simet al\.[2025](https://arxiv.org/html/2608.06908#bib.bib70)\), establishing a pathway from embedding\-level bias to changes in human implicit associations\. If intrinsic measurements are distorted by anisotropy, the resulting inaccuracies may lead to erroneous assessments of downstream risk, making geometric calibration a prerequisite for reliable bias auditing\.

As AI systems come under regulatory oversight, bias auditing tools must meet high standards of reliability\. The EU AI Act\(European Parliament and Council of the European Union[2024](https://arxiv.org/html/2608.06908#bib.bib60)\)mandates bias examination as part of its data governance requirements for datasets used in high\-risk AI systems, and the NIST AI Risk Management Framework\(National Institute of Standards and Technology[2023](https://arxiv.org/html/2608.06908#bib.bib61)\)emphasizes valid and reliable measurement in AI risk assessment\. ZCA calibration could be integrated into such frameworks as a standard preprocessing step for embedding\-based bias measurement, ensuring comparable audit results across models\.

### 6\.2Compatibility with EAT Variants

ZCA whitening operates as a pre\-processing calibration step that is agnostic to the specific form of the association test applied afterward\. Because it transforms only the embedding space geometry without modifying the test procedure itself, it is compatible with any cosine\-similarity\-based bias metric\. It can thus be applied to sentence embeddings in SEAT\(Mayet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib15)\), contextualized vectors in CEAT\(Guo and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib16)\), image embeddings in iEAT\(Steed and Caliskan[2021](https://arxiv.org/html/2608.06908#bib.bib17)\), and speech representations in SpEAT\(Slaughteret al\.[2023](https://arxiv.org/html/2608.06908#bib.bib18)\)\. For ML\-EAT\(Wolfeet al\.[2024](https://arxiv.org/html/2608.06908#bib.bib8)\), our approach provides a corrective step that complements its anisotropy diagnostic: when ML\-EAT identifies a model as highly anisotropic, ZCA whitening can restore the conditions under which cosine\-based evaluation becomes meaningful\. More broadly, any future variant of embedding association tests relying on cosine similarity would benefit from the same calibration procedure\.

### 6\.3Limitations

Several limitations should be acknowledged\. First, the whitening matrix is estimated from WikiText\-103, and different reference corpora may yield different transformations; the sensitivity of WEAT outcomes to the choice and size of the reference corpus warrants systematic investigation\. Second, our evaluation is limited to seven models, and whether the calibration behavior we observe generalizes to a broader range of models requires further investigation\. Third, our implementation uses a fixed regularization parameterε\\varepsilonfor numerical stability, and we do not provide a sensitivity analysis of how this choice affects calibrated WEAT scores\. Fourth, the relationship between calibrated WEAT measurements and downstream task bias remains an open question\. Prior work has reported mixed findings:Goldfarb\-Tarrantet al\.\([2021](https://arxiv.org/html/2608.06908#bib.bib56)\)found frequent divergence, whileGhateet al\.\([2025](https://arxiv.org/html/2608.06908#bib.bib62)\)demonstrated correlation in vision\-language models\. Whether calibration improves this relationship requires future investigation\. Addressing these open questions constitutes an important direction for future research\.

Beyond these limitations, we emphasize the scope of our contribution\. This work evaluates the*geometric reliability*of WEAT measurements\. We examine whether cosine similarity is a valid metric within the embedding space, not how the associations it measures shape the outputs of a deployed system\. We do not evaluate the alignment between WEAT measurements and human implicit association measures\. The latter is a separate question that applies equally to raw and calibrated measurements and has been examined in prior work\(Caliskanet al\.[2017](https://arxiv.org/html/2608.06908#bib.bib2); Morehouseet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib72); Charlesworthet al\.[2024a](https://arxiv.org/html/2608.06908#bib.bib73)\)\. Extending human alignment analysis to calibrated measurements is a natural next step\.

## 7Conclusion

This research proposed ZCA whitening as a pre\-processing calibration step for the Word Embedding Association Test \(WEAT\)\. The method transforms the covariance of the embedding space into the identity matrix while minimizing perturbation to the original vectors, restoring the isotropy condition on which WEAT relies\. We evaluated our approach on ten WEAT test suites and seven models \(70 model–task combinations\)\. ZCA whitening reduced anisotropy across all models, and for highly anisotropic models it also improved standard semantic similarity benchmark scores\. After calibration, over 30% of WEAT results changed significance status, and effect sizes shifted in both directions depending on bias category\.

These results suggest that bias measurements obtained in strongly anisotropic spaces may partly reflect geometric properties of the space rather than social associations alone\. We do not claim that previously reported findings are invalid\. Rather, studies conducted under such conditions may benefit from re\-examination with a calibrated metric\.

ZCA whitening operates as a model\-agnostic pre\-processing step\. It can be applied not only to WEAT but also to its variants, such as CEAT, iEAT, and SpEAT, and in principle to any bias metric that relies on cosine similarity\. The reliability of such measurements is inseparable from the geometry of the space in which they are computed, and our approach offers a practical step toward geometrically valid bias measurement in computational social science and AI fairness research\.

## Ethical Statement

This work aims to improve the reliability of bias measurement in word embeddings\. However, several ethical considerations deserve attention\.

##### Risk of misinterpretation\.

Our finding that some previously significant bias measurements may not survive geometric calibration could be misinterpreted as evidence that AI systems are less biased than previously reported\. We caution against this interpretation\. Calibration reveals that while some specific measurements were inflated, others were deflated, and the overall picture is one of measurement unreliability rather than bias absence\. The appropriate response is to re\-measure with calibrated tools, not to conclude that bias is less prevalent\. This concern is particularly acute given the bidirectional nature of the distortion we document: practitioners, journalists, or model developers who selectively cite Disappearing cases as evidence of model improvement would be misrepresenting our findings\. AsBlodgettet al\.\([2020](https://arxiv.org/html/2608.06908#bib.bib63)\)argued, NLP bias research must be grounded in clear normative reasoning about who is harmed and how; our calibration method addresses one specific source of measurement error but does not resolve deeper questions about what constitutes bias or how bias measurements should inform practice\.

##### Scope of intrinsic bias metrics\.

Even with proper geometric calibration, intrinsic bias measurements capture only one dimension of the complex ways in which AI systems can perpetuate social inequities\. Embedding\-level bias metrics should be used alongside downstream task evaluations\(Goldfarb\-Tarrantet al\.[2021](https://arxiv.org/html/2608.06908#bib.bib56); Cabelloet al\.[2023](https://arxiv.org/html/2608.06908#bib.bib10)\), qualitative audits, and participatory assessments involving affected communities\(Selbstet al\.[2019](https://arxiv.org/html/2608.06908#bib.bib64)\)\. The communities most likely to be harmed by biased AI systems include racial, gender, and other minoritized groups represented in WEAT target sets\. Their perspectives should inform what counts as a meaningful bias measurement in the first place\.

##### Responsibility in auditing contexts\.

As discussed in Section[6\.1](https://arxiv.org/html/2608.06908#S6.SS1), embedding\-based bias measurements are increasingly used in regulatory and auditing contexts\. This dual\-use character introduces specific responsibilities: a calibration method that changes which biases are flagged as significant could, if applied uncritically, either expose previously hidden harms or obscure documented ones\. We therefore recommend that calibrated measurements be reported alongside uncalibrated ones rather than replacing them, so that audit trails preserve both perspectives and allow stakeholders to assess the geometric reliability of each measurement\.

## Acknowledgments

We thank the reviewers of AIES 2026 for their thoughtful and constructive feedback\. We also thank our colleagues for helpful discussions throughout this work\.

## References

- A\. H\. Bailey, A\. Williams, and A\. Cimpian \(2022\)Based on billions of words on the internet, people = men\.Science Advances8\(13\),pp\. eabm2463\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- S\. L\. Blodgett, S\. Barocas, H\. Daumé III, and H\. Wallach \(2020\)Language \(technology\) is power: a critical survey of “bias” in NLP\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5454–5476\.External Links:[Link](https://aclanthology.org/2020.acl-main.485/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.485)Cited by:[Risk of misinterpretation\.](https://arxiv.org/html/2608.06908#Sx1.SS0.SSS0.Px1.p1.1)\.
- T\. Bolukbasi, K\. Chang, J\. Y\. Zou, V\. Saligrama, and A\. T\. Kalai \(2016\)Man is to computer programmer as woman is to homemaker? debiasing word embeddings\.Advances in Neural Information Processing Systems29\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1)\.
- R\. Bommasani, K\. Davis, and C\. Cardie \(2020\)Interpreting pretrained contextualized representations via reductions to static embeddings\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 4758–4781\.Cited by:[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px4.p1.3)\.
- N\. Borenstein, K\. Stańczak, T\. Rolskov, N\. K\. Käfer, N\. da Silva Perez, and I\. Augenstein \(2023\)Measuring intersectional biases in historical documents\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 2711–2730\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- L\. Cabello, A\. K\. Jørgensen, and A\. Søgaard \(2023\)On the independence of association bias and empirical fairness in language models\.InProceedings of the 2023 ACM conference on fairness, accountability, and transparency,pp\. 370–378\.Cited by:[Scope of intrinsic bias metrics\.](https://arxiv.org/html/2608.06908#Sx1.SS0.SSS0.Px2.p1.1)\.
- X\. Cai, J\. Huang, Y\. Bian, and K\. Church \(2021\)Isotropy in the contextual embedding space: Clusters and manifolds\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1)\.
- A\. Caliskan, P\. P\. Ajay, T\. Charlesworth, R\. Wolfe, and M\. R\. Banaji \(2022\)Gender bias in word embeddings: a comprehensive analysis of frequency, syntax, and semantics\.InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 156–170\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- A\. Caliskan, J\. J\. Bryson, and A\. Narayanan \(2017\)Semantics derived automatically from language corpora contain human\-like biases\.Science356\(6334\),pp\. 183–186\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1),[§1](https://arxiv.org/html/2608.06908#S1.p6.1),[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p1.5),[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.06908#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2608.06908#S4.T1),[§6\.3](https://arxiv.org/html/2608.06908#S6.SS3.p2.1)\.
- D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. Specia \(2017\)SemEval\-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation\.InProceedings of the 11th International Workshop on Semantic Evaluation \(SemEval\-2017\),Vancouver, Canada,pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.18653/v1/S17-2001),[Link](https://aclanthology.org/S17-2001/)Cited by:[§4\.3](https://arxiv.org/html/2608.06908#S4.SS3.SSS0.Px2.p1.1)\.
- T\. E\. S\. Charlesworth, K\. Morehouse, V\. Rouduri, and W\. A\. Cunningham \(2024a\)Echoes of culture: relationships of implicit and explicit attitudes with contemporary English, historical English, and 53 non\-English languages\.Social Psychological and Personality Science15\(7\),pp\. 812–823\.External Links:[Document](https://dx.doi.org/10.1177/19485506241256400)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1),[§6\.3](https://arxiv.org/html/2608.06908#S6.SS3.p2.1)\.
- T\. E\. Charlesworth, A\. Caliskan, and M\. R\. Banaji \(2022\)Historical representations of social groups across 200 years of word embeddings from google books\.Proceedings of the National Academy of Sciences119\(28\),pp\. e2121798119\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p2.1)\.
- T\. E\. Charlesworth, K\. Ghate, A\. Caliskan, and M\. R\. Banaji \(2024b\)Extracting intersectional stereotypes from embeddings: developing and validating the flexible intersectional stereotype extraction procedure\.PNAS Nexus3\(3\),pp\. pgae089\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p2.1)\.
- T\. E\. Charlesworth, V\. Yang, T\. C\. Mann, B\. Kurdi, and M\. R\. Banaji \(2021\)Gender stereotypes in natural language: word embeddings show robust consistency across child and adult language corpora of more than 65 million words\.Psychological Science32\(2\),pp\. 218–240\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- J\. Cobert, H\. Mills, A\. Lee, O\. Gologorskaya, E\. Espejo, S\. Y\. Jeon, W\. J\. Boscardin, T\. A\. Heintz, C\. J\. Kennedy, D\. C\. Ashana, A\. C\. Chapman, and S\. J\. Lee \(2024\)Measuring implicit bias in ICU notes using word\-embedding neural network models\.Chest165\(6\),pp\. 1481–1490\.External Links:[Document](https://dx.doi.org/10.1016/j.chest.2023.12.031)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- J\. Cohen \(1992\)A power primer\.Psychological Bulletin112\(1\),pp\. 155–159\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p1.8)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2)\.
- S\. Dutta, P\. Srivastava, V\. Solunke, S\. Nath, and A\. R\. KhudaBukhsh \(2023\)Disentangling societal inequality from model biases: gender inequality in divorce court proceedings\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence \(IJCAI\-23\),pp\. 5959–5967\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2023/661)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- K\. Ethayarajh \(2019\)How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt\-2 embeddings\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 55–65\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p2.1),[§1](https://arxiv.org/html/2608.06908#S1.p3.1),[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.06908#S3.SS1.p1.2)\.
- European Parliament and Council of the European Union \(2024\)Regulation \(EU\) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence \(AI Act\)\.Note:Official Journal of the European Union, L 1689\.https://eur\-lex\.europa\.eu/eli/reg/2024/1689/ojAccessed: 2026\-08\-04Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p6.1)\.
- L\. Finkelstein, E\. Gabrilovich, Y\. Matias, E\. Rivlin, Z\. Solan, G\. Wolfman, and E\. Ruppin \(2002\)Placing search in context: The concept revisited\.ACM Transactions on Information Systems20\(1\),pp\. 116–131\.Cited by:[§4\.3](https://arxiv.org/html/2608.06908#S4.SS3.SSS0.Px2.p1.1)\.
- F\. Friedrich, M\. Brack, L\. Struppek, D\. Hintersdorf, P\. Schramowski, S\. Luccioni, and K\. Kersting \(2025\)Auditing and instructing text\-to\-image generation models on fairness\.AI and Ethics5\(3\),pp\. 2103–2123\.External Links:[Document](https://dx.doi.org/10.1007/s43681-024-00531-5)Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1)\.
- T\. Gao, X\. Yao, and D\. Chen \(2021\)SimCSE: simple contrastive learning of sentence embeddings\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 6894–6910\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.552/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.552)Cited by:[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px3.p1.1),[§5\.1](https://arxiv.org/html/2608.06908#S5.SS1.p1.1)\.
- N\. Garg, L\. Schiebinger, D\. Jurafsky, and J\. Zou \(2018\)Word embeddings quantify 100 years of gender and ethnic stereotypes\.Proceedings of the National Academy of Sciences115\(16\),pp\. E3635–E3644\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p2.1)\.
- K\. Ghate, T\. Charlesworth, M\. T\. Diab, and A\. Caliskan \(2025\)Biases propagate in encoder\-based vision\-language models: a systematic analysis from intrinsic measures to zero\-shot retrieval outcomes\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18562–18580\.External Links:[Link](https://aclanthology.org/2025.findings-acl.955/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.955),ISBN 979\-8\-89176\-256\-5Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1),[§6\.3](https://arxiv.org/html/2608.06908#S6.SS3.p1.1)\.
- N\. Godey, É\. de la Clergerie, and B\. Sagot \(2024\)Anisotropy is inherent to self\-attention in transformers\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 35–48\.External Links:[Link](https://aclanthology.org/2024.eacl-long.3/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.3)Cited by:[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1)\.
- S\. Goldfarb\-Tarrant, R\. Marchant, R\. Muñoz Sánchez, M\. Pandya, and A\. Lopez \(2021\)Intrinsic bias metrics do not correlate with application bias\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 1926–1940\.Cited by:[§6\.3](https://arxiv.org/html/2608.06908#S6.SS3.p1.1),[Scope of intrinsic bias metrics\.](https://arxiv.org/html/2608.06908#Sx1.SS0.SSS0.Px2.p1.1)\.
- M\. Gray and L\. Wu \(2025\)Benchmarking bias in embeddings of healthcare AI models: using SD\-WEAT for detection and measurement across sensitive populations\.BMC Medical Informatics and Decision Making25\(1\),pp\. 258\.External Links:[Document](https://dx.doi.org/10.1186/s12911-025-03102-8)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- A\. G\. Greenwald, D\. E\. McGhee, and J\. L\. Schwartz \(1998\)Measuring individual differences in implicit cognition: the implicit association test\.\.Journal of personality and social psychology74\(6\),pp\. 1464\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p1.5),[§5\.3](https://arxiv.org/html/2608.06908#S5.SS3.SSSx1.p1.3)\.
- A\. G\. Greenwald, M\. Brendl, H\. Cai, D\. Cvencek, J\. F\. Dovidio, M\. Friese, A\. Hahn, E\. Hehman, W\. Hofmann, S\. Hughes, I\. Hussey, C\. Jordan, T\. A\. Kirby, C\. K\. Lai, J\. W\. B\. Lang, K\. P\. Lindgren, D\. Maison, B\. D\. Ostafin, J\. R\. Rae, K\. A\. Ratliff, A\. Spruyt, and R\. W\. Wiers \(2022\)Best research practices for using the Implicit Association Test\.Behavior Research Methods54\(3\),pp\. 1161–1180\.External Links:[Document](https://dx.doi.org/10.3758/s13428-021-01624-3)Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p4.1)\.
- W\. Guo and A\. Caliskan \(2021\)Detecting emergent intersectional biases: contextualized word embeddings contain a distribution of human\-like biases\.InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 122–133\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2),[§4\.2](https://arxiv.org/html/2608.06908#S4.SS2.p1.1),[§6\.2](https://arxiv.org/html/2608.06908#S6.SS2.p1.1)\.
- F\. Hill, R\. Reichart, and A\. Korhonen \(2015\)SimLex\-999: Evaluating semantic models with \(genuine\) similarity estimation\.Computational Linguistics41\(4\),pp\. 665–695\.Cited by:[§4\.3](https://arxiv.org/html/2608.06908#S4.SS3.SSS0.Px2.p1.1)\.
- J\. Huang, D\. Tang, W\. Zhong, S\. Lu, L\. Shou, M\. Gong, D\. Jiang, and N\. Duan \(2021\)WhiteningBERT: an easy unsupervised sentence embedding approach\.InFindings of the association for computational linguistics: EMNLP 2021,pp\. 238–244\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p5.1),[§3\.2](https://arxiv.org/html/2608.06908#S3.SS2.p4.1)\.
- A\. Kessy, A\. Lewin, and K\. Strimmer \(2018\)Optimal whitening and decorrelation\.The American Statistician72\(4\),pp\. 309–314\.Cited by:[§3\.2](https://arxiv.org/html/2608.06908#S3.SS2.p4.1)\.
- K\. Kurita, N\. Vyas, A\. Pareek, A\. W\. Black, and Y\. Tsvetkov \(2019\)Measuring bias in contextualized word representations\.InProceedings of the first workshop on gender bias in natural language processing,pp\. 166–172\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.06908#S4.SS2.p1.1)\.
- S\. Leach, A\. P\. Kitchin, and R\. M\. Sutton \(2023\)Word embeddings reveal growing moral concern for people, animals and the environment\.British Journal of Social Psychology62\(4\),pp\. 1925–1938\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- M\. Lewis and G\. Lupyan \(2020\)Gender stereotypes are reflected in the distributional structure of 25 languages\.Nature Human Behaviour4\(10\),pp\. 1021–1028\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p2.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: A robustly optimized BERT pretraining approach\.External Links:1907\.11692Cited by:[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2)\.
- A\. Machina and R\. Mercer \(2024\)Anisotropy is not inherent to transformers\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 4892–4907\.External Links:[Link](https://aclanthology.org/2024.naacl-long.274/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.274)Cited by:[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1)\.
- T\. Manzini, L\. Yao Chong, A\. W\. Black, and Y\. Tsvetkov \(2019\)Black is to criminal as Caucasian is to police: detecting and removing multiclass bias in word embeddings\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 615–621\.External Links:[Link](https://aclanthology.org/N19-1062/),[Document](https://dx.doi.org/10.18653/v1/N19-1062)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- S\. Matthews, J\. Hudzina, and D\. Sepehr \(2022\)Gender and racial stereotype detection in legal opinion word embeddings\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 12026–12033\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- C\. May, A\. Wang, S\. Bordia, S\. Bowman, and R\. Rudinger \(2019\)On measuring social biases in sentence encoders\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 622–628\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2),[§4\.2](https://arxiv.org/html/2608.06908#S4.SS2.p1.1),[§6\.2](https://arxiv.org/html/2608.06908#S6.SS2.p1.1)\.
- K\. Mei, S\. Fereidooni, and A\. Caliskan \(2023\)Bias against 93 stigmatized groups in masked language models and downstream sentiment classification tasks\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency \(FAccT ’23\),Chicago, IL, USA,pp\. 1699–1710\.External Links:[Document](https://dx.doi.org/10.1145/3593013.3594109)Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InProceedings of the 5th International Conference on Learning Representations \(ICLR 2017\),Cited by:[§3\.3](https://arxiv.org/html/2608.06908#S3.SS3.p2.3),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px4.p1.3)\.
- T\. Mikolov, I\. Sutskever, K\. Chen, G\. Corrado, and J\. Dean \(2013\)Distributed representations of words and phrases and their compositionality\.InAdvances in Neural Information Processing Systems,Vol\.26\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px1.p1.1)\.
- K\. N\. Morehouse, T\. E\. S\. Charlesworth, V\. Rouduri, and W\. A\. Cunningham \(2023\)Traces of human attitudes in contemporary and historical word embeddings \(1800–2000\)\.Research Square\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.21203/rs.3.rs-2922677/v1),[Link](https://www.researchsquare.com/article/rs-2922677/v1)Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1),[§6\.3](https://arxiv.org/html/2608.06908#S6.SS3.p2.1)\.
- K\. Morehouse, S\. Swaroop, and W\. Pan \(2025\)Position: rethinking LLM bias probing using lessons from the social sciences\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 81841–81860\.Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p4.1)\.
- J\. Mu and P\. Viswanath \(2018\)All\-but\-the\-Top: Simple and effective post\-processing for word representations\.InProceedings of the 6th International Conference on Learning Representations \(ICLR 2018\),Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p3.1),[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p2.1),[§3\.1](https://arxiv.org/html/2608.06908#S3.SS1.p2.2)\.
- A\. Mukherjee, C\. Raj, Z\. Zhu, and A\. Anastasopoulos \(2023\)Global voices, local biases: socio\-cultural prejudices across languages\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 15828–15845\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- C\. Napp \(2023\)Gender stereotypes embedded in natural language are stronger in more economically developed and individualistic countries\.PNAS Nexus2\(11\),pp\. pgad355\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- National Institute of Standards and Technology \(2023\)Artificial intelligence risk management framework \(AI RMF 1\.0\)\.Technical reportTechnical ReportNIST AI 100\-1,National Institute of Standards and Technology\.Note:https://doi\.org/10\.6028/NIST\.AI\.100\-1Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p6.1)\.
- J\. Pennington, R\. Socher, and C\. D\. Manning \(2014\)GloVe: Global vectors for word representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing,pp\. 1532–1543\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2)\.
- S\. Rajaee and M\. T\. Pilehvar \(2022\)An isotropy analysis in the multilingual BERT embedding space\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1309–1316\.External Links:[Link](https://aclanthology.org/2022.findings-acl.103/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.103)Cited by:[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1)\.
- A\. Rios, R\. Joshi, and H\. Shin \(2020\)Quantifying 60 years of gender bias in biomedical research with word embeddings\.InProceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing \(BioNLP 2020\),pp\. 1–13\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p2.1)\.
- C\. Ross, B\. Katz, and A\. Barbu \(2021\)Measuring social biases in grounded vision and language embeddings\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 998–1008\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1)\.
- W\. Rudman, N\. Gillman, T\. Rayne, and C\. Eickhoff \(2022\)IsoScore: measuring the uniformity of embedding space utilization\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3325–3339\.External Links:[Link](https://aclanthology.org/2022.findings-acl.262/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.262)Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.06908#S3.SS1.p1.2)\.
- K\. G\. Schmahl, T\. J\. Viering, S\. Makrodimitris, A\. N\. Jahfari, D\. Tax, and M\. Loog \(2020\)Is wikipedia succeeding in reducing gender bias? assessing changes in gender bias in wikipedia using word embeddings\.InProceedings of the fourth workshop on natural language processing and computational social science,pp\. 94–103\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- A\. D\. Selbst, D\. Boyd, S\. A\. Friedler, S\. Venkatasubramanian, and J\. Vertesi \(2019\)Fairness and abstraction in sociotechnical systems\.InProceedings of the Conference on Fairness, Accountability, and Transparency,pp\. 59–68\.Cited by:[Scope of intrinsic bias metrics\.](https://arxiv.org/html/2608.06908#Sx1.SS0.SSS0.Px2.p1.1)\.
- M\. Sim, N\. G\. Brigham, T\. Kohno, T\. E\. S\. Charlesworth, and A\. Caliskan \(2025\)Biased AI outputs can impact humans’ implicit bias: a case study of the impact of gender\-biased text\-to\-image generators\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society \(AIES\),Vol\.8,pp\. 2375–2386\.Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1)\.
- I\. Slaughter, C\. Greenberg, R\. Schwartz, and A\. Caliskan \(2023\)Pre\-trained speech processing models contain human\-like biases that propagate to speech emotion recognition\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 8967–8989\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2608.06908#S6.SS2.p1.1)\.
- R\. Steed and A\. Caliskan \(2021\)Image representations learned with unsupervised pre\-training contain human\-like biases\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 701–713\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§6\.2](https://arxiv.org/html/2608.06908#S6.SS2.p1.1)\.
- J\. Su, J\. Cao, W\. Liu, and Y\. Ou \(2021\)Whitening sentence representations for better semantics and faster retrieval\.External Links:2103\.15316Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p5.1),[§3\.2](https://arxiv.org/html/2608.06908#S3.SS2.p4.1)\.
- Y\. Sun, M\. Saxon, I\. Yang, A\. Gueorguieva, and A\. Caliskan \(2026\)VEAT quantifies implicit associations in text\-to\-video generator Sora and reveals challenges in bias mitigation\.External Links:2601\.00996Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1)\.
- C\. Sunsay \(2023\)A historical evaluation of the disease avoidance theory of xenophobia\.PLOS ONE18\(12\),pp\. e0294816\.Cited by:[§2\.2](https://arxiv.org/html/2608.06908#S2.SS2.p1.1)\.
- W\. Timkey and M\. Van Schijndel \(2021\)All bark and no bite: rogue dimensions in transformer language models obscure representational quality\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 4527–4546\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p2.1),[§1](https://arxiv.org/html/2608.06908#S1.p3.1),[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.06908#S3.SS1.p2.2),[§5\.1](https://arxiv.org/html/2608.06908#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2608.06908#S5.SS2.p1.8)\.
- R\. Wolfe, M\. R\. Banaji, and A\. Caliskan \(2022a\)Evidence for hypodescent in visual semantic ai\.InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency,pp\. 1293–1304\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1)\.
- R\. Wolfe, M\. R\. Banaji, and A\. Caliskan \(2022b\)Markedness in visual semantic AI\.InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency,pp\. 1269–1279\.Cited by:[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1)\.
- R\. Wolfe and A\. Caliskan \(2022\)American==White in Multimodal Language\-and\-Image AI\.InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 800–812\.Cited by:[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§6\.1](https://arxiv.org/html/2608.06908#S6.SS1.p5.1)\.
- R\. Wolfe, A\. Hiniker, and B\. Howe \(2024\)ML\-EAT: a multilevel embedding association test for interpretable and transparent social science\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.7,pp\. 1608–1620\.Cited by:[§1](https://arxiv.org/html/2608.06908#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.06908#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p2.1),[§3\.1](https://arxiv.org/html/2608.06908#S3.SS1.p1.2),[§4\.1](https://arxiv.org/html/2608.06908#S4.SS1.SSS0.Px2.p1.2),[§4\.2](https://arxiv.org/html/2608.06908#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2608.06908#S5.SS1.p1.1),[§6\.2](https://arxiv.org/html/2608.06908#S6.SS2.p1.1)\.
- K\. Zhou, K\. Ethayarajh, D\. Card, and D\. Jurafsky \(2022\)Problems with cosine as a measure of embedding similarity for high frequency words\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Dublin, Ireland,pp\. 401–423\.Cited by:[§2\.3](https://arxiv.org/html/2608.06908#S2.SS3.p1.1)\.

Supplementary Material for: Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre\-Processing Step for Embedding Association Tests

## Appendix ADerivation of the Identity Covariance Property of ZCA\-Whitened Vectors

This appendix provides the complete derivation showing that the ZCA\-whitened vectorszi=WZCA​\(xi−μ\)z\_\{i\}=W\_\{\\text\{ZCA\}\}\(x\_\{i\}\-\\mu\)have identity covariance, as summarized in Equation 10 of the main paper\.

Starting from the definition of the sample covariance of\{zi\}\\\{z\_\{i\}\\\}:

Cov⁡\(z\)\\displaystyle\\operatorname\{Cov\}\(z\)=1n−1​∑i=1nzi​zi⊤\\displaystyle=\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}z\_\{i\}z\_\{i\}^\{\\top\}=1n−1​∑i=1nWZCA​\(xi−μ\)​\(WZCA​\(xi−μ\)\)⊤\\displaystyle=\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}W\_\{\\text\{ZCA\}\}\(x\_\{i\}\-\\mu\)\\bigl\(W\_\{\\text\{ZCA\}\}\(x\_\{i\}\-\\mu\)\\bigr\)^\{\\top\}=1n−1​∑i=1nWZCA​\(xi−μ\)​\(xi−μ\)⊤​WZCA⊤\\displaystyle=\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}W\_\{\\text\{ZCA\}\}\(x\_\{i\}\-\\mu\)\(x\_\{i\}\-\\mu\)^\{\\top\}W\_\{\\text\{ZCA\}\}^\{\\top\}=WZCA​\(1n−1​∑i=1n\(xi−μ\)​\(xi−μ\)⊤\)​WZCA⊤\\displaystyle=W\_\{\\text\{ZCA\}\}\\left\(\\frac\{1\}\{n\-1\}\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\mu\)\(x\_\{i\}\-\\mu\)^\{\\top\}\\right\)W\_\{\\text\{ZCA\}\}^\{\\top\}=WZCA​Σ​WZCA⊤\\displaystyle=W\_\{\\text\{ZCA\}\}\\,\\Sigma\\,W\_\{\\text\{ZCA\}\}^\{\\top\}=Σ−1/2​Σ​Σ−1/2\\displaystyle=\\Sigma^\{\-1/2\}\\,\\Sigma\\,\\Sigma^\{\-1/2\}=I\.\\displaystyle=I\.\(A1\)The third equality uses the transpose identity\(A​B\)⊤=B⊤​A⊤\(AB\)^\{\\top\}=B^\{\\top\}A^\{\\top\}\. The sixth equality substitutesWZCA=Σ−1/2W\_\{\\text\{ZCA\}\}=\\Sigma^\{\-1/2\}, using the fact that this matrix is symmetric \(soWZCA⊤=WZCAW\_\{\\text\{ZCA\}\}^\{\\top\}=W\_\{\\text\{ZCA\}\}\)\. The final equality follows from

Σ−1/2​Σ​Σ−1/2=Σ−1/2​Σ1/2​Σ1/2​Σ−1/2=I\.\\Sigma^\{\-1/2\}\\,\\Sigma\\,\\Sigma^\{\-1/2\}=\\Sigma^\{\-1/2\}\\,\\Sigma^\{1/2\}\\,\\Sigma^\{1/2\}\\,\\Sigma^\{\-1/2\}=I\.

## Appendix BPairwise Cosine Similarity Heatmaps for All Models

Figures[A1](https://arxiv.org/html/2608.06908#A2.F1)–[A3](https://arxiv.org/html/2608.06908#A2.F3)show pairwise cosine similarity matrices for 2,000 randomly sampled embedding vectors, grouped by architectural family\. Across all seven models, ZCA whitening substantially reduces pairwise similarity regardless of the initial anisotropy severity\.

![Refer to caption](https://arxiv.org/html/2608.06908v1/x5.png)Figure A1:Pairwise cosine similarity matrices for static embedding models \(GloVe, word2vec\) before \(left\) and after \(right\) ZCA whitening\.![Refer to caption](https://arxiv.org/html/2608.06908v1/x6.png)Figure A2:Pairwise cosine similarity matrices for contextualized models \(BERT, RoBERTa, GPT\-2\) before \(left\) and after \(right\) ZCA whitening\.![Refer to caption](https://arxiv.org/html/2608.06908v1/x7.png)Figure A3:Pairwise cosine similarity matrices for contrastive models \(unsup\-BERT, unsup\-RoBERTa\) before \(left\) and after \(right\) ZCA whitening\.
## Appendix CFull WEAT Results

Table A1:WEAT effect sizes \(dd\) and permutationpp\-values \(100,000 permutations, one\-sided\) before \(Raw\) and after \(White\) ZCA whitening across seven models and ten test suites \(see Table 1 for full definitions\)\. Significance atp<0\.05p<0\.05\.S= Stable,D= Disappearing \(significant→\\tonon\-significant\),E= Emerging \(non\-significant→\\tosignificant\)\. Boldddvalues indicatep<0\.05p<0\.05\.pp\-values smaller than10−410^\{\-4\}are reported as their order of magnitude\.Static Embeddings

Contextualized Embeddings

Unsupervised Contrastive Embeddings