Speaker Group Encoding in Self-supervised Speech Recognition Models

# Speaker Group Encoding in Self-supervised Speech Recognition Models
Source: [https://arxiv.org/html/2606.10654](https://arxiv.org/html/2606.10654)
Felix Herron1,2&Solange Rossato2&Alexandre Allauzen1&Benoit Favre2,3&François Portet2 MILES Team, LAMSADE, Université Paris Dauphine\-PSL, France \(1\) GETALP Team, LIG, Université Grenoble Alpes, France \(2\) NLP team, LIS, Aix\-Marseille University, France \(3\) felix\.herron@daupine\.eu

###### Abstract

We investigate what self\-supervised speech recognition models \(S3Ms\) learn about speaker groups \(SGs\)\. We examine several states of S3Ms: pretrained, finetuned on speaker identification \(SID\), finetuned on automatic speech recognition \(ASR\), and ASR\-finetuned using a fairness enhancing algorithm\. We find that S3Ms encode information about several speaker group categories \(SGCs\), including their gender, age, dialect, ethnicity, and whether they are a native speaker\. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature\. On the other hand, finetuning for ASR discards phonetically variant speaker group information \(SGI\) but retains semantically variant SGI\. We find that ASR algorithms designed for fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs\. We discuss how SGI is encoded by each layer, and identify subdimensions of embeddings responsible for encoding different SGCs\. Finally, we discuss how our findings could be beneficial in designing fairer ASR algorithms\.

## 1Introduction

Self\-supervised speech encoder models \(S3Ms\) are popular backbone tools for many speech\-related downstream tasks\. They are particularly versatile because they learn task\-agnostic embeddings during pretraining, which requires pure audio data only\. The rich embeddings learned during pretraining can be paired with lightweight downstream models to finetune an S3M to specific downstream tasks such as automatic speech recognition \(ASR\) or speaker identification \(SID\), among many others\[[30](https://arxiv.org/html/2606.10654#bib.bib116)\]\. The lightweight downstream models learn to map from the latent output space of S3Ms to a downstream task space\.

There has been considerable study showing that S3Ms perform better on downstream tasks for certain speaker groups \(SGs\) than others\. For example, non\-native speakers are shown to be understood less well by ASR systems\[[10](https://arxiv.org/html/2606.10654#bib.bib123),[29](https://arxiv.org/html/2606.10654#bib.bib79),[23](https://arxiv.org/html/2606.10654#bib.bib112)\]; the same is true of non\-standard dialects of various languages\[[10](https://arxiv.org/html/2606.10654#bib.bib123),[23](https://arxiv.org/html/2606.10654#bib.bib112),[13](https://arxiv.org/html/2606.10654#bib.bib118),[14](https://arxiv.org/html/2606.10654#bib.bib98),[13](https://arxiv.org/html/2606.10654#bib.bib118)\], as well as very young and very old speakers\[[10](https://arxiv.org/html/2606.10654#bib.bib123),[23](https://arxiv.org/html/2606.10654#bib.bib112)\]\. These observations support the hypothesis that S3Ms model utterances of different SGs differently\. This paper addresses the question as to how these differences are manifested in an S3M’s layer\-by\-layer processing of an utterance\. We present analyses of which types of speaker group information \(SGI\) are present at each layer for various model architectures, and how this SGI presence shifts as models are finetuned\. We also analyze the neuron\-level representations of SGI, and discuss how SGI is distributed across latent embedding dimensions\. Our paper is focused on interpretability of S3Ms, and while we don’t provide a concrete fairness promoting algorithm based on our work, we suggest potential avenues for future work in fairness promotion based on our results\.

## 2Related work

### 2\.1S3M interpretability

The nature of transformer\-based speech S3M embeddings has been studied since their inception in 2021\[[2](https://arxiv.org/html/2606.10654#bib.bib135)\]\. For example, they are used in benchmarks such as SUPERB in English or LeBenchmark in French, where they have been shown able to solve any number of speech related tasks with or without finetuning\[[30](https://arxiv.org/html/2606.10654#bib.bib116),[8](https://arxiv.org/html/2606.10654#bib.bib72)\]\. In\[[3](https://arxiv.org/html/2606.10654#bib.bib137)\], authors analyze which layers perform best on a variety of downstream tasks, showing that earlier layers tend to contain more information on speaker identity related tasks \(such as SID or speaker diarization\), while later layers are better suited for more semantic tasks \(such as ASR or intent classification\)\. Building on this,\[[18](https://arxiv.org/html/2606.10654#bib.bib124)\]and\[[22](https://arxiv.org/html/2606.10654#bib.bib5)\]show that phonetic information peaks in middle layers and word identity information in later layers\. Furthermore, they showed S3Ms can be viewed as autoencoders, with later layers learning features similar to initial layers \(logical given reconstructive S3M pretraining objectives, generally cross\-entropy or contrastive loss for masked input subsequences\[[28](https://arxiv.org/html/2606.10654#bib.bib20)\]\), though this trend disappears upon ASR finetuning\.\[[19](https://arxiv.org/html/2606.10654#bib.bib125)\]shows that S3Ms which use CNNs as acoustic feature extractors \(rather than using Mel filterbank coefficients \(MFCs\), for example\), tend to learn a representation similar to MFCs anyway\. They also show that, depending on pretraining objective, different models store phone/word information in different layers\.\[[4](https://arxiv.org/html/2606.10654#bib.bib97)\]shows that S3Ms encode some semantic information of word meaning, though significantly less than they do raw phonetic content\.

There has also been work on speaker\-level layerwise interpretability\. For example,\[[32](https://arxiv.org/html/2606.10654#bib.bib8)\]shows that S3Ms finetuned for ASR achieve high SID performance for early layers, but lose it almost entirely in final layers\. Furthermore,\[[15](https://arxiv.org/html/2606.10654#bib.bib93)\]shows that speaker information is stored orthogonally from phoneme information in intermediate S3M representations, and\[[16](https://arxiv.org/html/2606.10654#bib.bib35)\]provides a method to quantitatively compare the orthogonality of types of information \(such as speaker information vs phoneme information\)\.\[[9](https://arxiv.org/html/2606.10654#bib.bib105)\]shows that the silent portions of S3M embeddings contain more speaker information than the sections with actual speech, and that adding silence artificially to an utterance boosts an S3M’s ability to model the speaker of that utterance\.

We emphasize one key difference between SID vs SG identification \(SGID\) \- the former involves learning to recognize a fixed set of speakers, while the latter must generalize to unknown speakers\. With this in mind, high performing SID does not necessarily generalize to high performing SGID\. There has been comparatively little work in SGID \- many publicly available speech corpora contain either few or inconsistently labeled metadata categories, so it is challenging to study\. One case where there has been some work is in Arabic dialect identification\[[24](https://arxiv.org/html/2606.10654#bib.bib14)\]\. This work is thus somewhat related to ours, though there is a far greater distance between different dialects of Arabic than dialects of English, and this distance is often heavily word\-based as opposed to phonetic\[[1](https://arxiv.org/html/2606.10654#bib.bib106)\]\.

## 3Methods

An S3M processes an utterance as a sequence of discretized acoustic embeddings corresponding to its length, and produces a corresponding sequence of rich embeddings, one for each discretized unit\.\[[26](https://arxiv.org/html/2606.10654#bib.bib82)\]show that taking the mean over all embeddings of an utterance produces a fixed\-length vector that contains information related to the speaker’s identity\. This is logical, as an average over all embeddings should cancel out local phonetic information and amplify global utterance\-level information\. We experiment with several pooling strategies, similarly to\[[9](https://arxiv.org/html/2606.10654#bib.bib105)\], by taking the mean over a subsequence of frame embeddings\. We experiment with five subsequences: the first frame, first 50 frames \(corresponding to one second of audio\), final frame, final 50 frames \(i\.e\. the final second of audio\), and the entire sequence\. We calculate these pooled embeddings for each model at each layer\.

In order to test for SGI, we first experimented with linear probes, as was proposed in the SUPERB benchmark for the SID task\[[30](https://arxiv.org/html/2606.10654#bib.bib116)\]\. These probes take the form of single linear layers:MS​G​C∈ℝdM,\|S​G​C\|M\_\{SGC\}\\in\\mathbb\{R\}^\{d\_\{M\},\|SGC\|\}, wheredMd\_\{M\}is the size of each model embedding \(i\.e\. 1024 for thelargeS3M configuration\), and\|S​G​C\|\|SGC\|is the number of classes in the SGC \(i\.e\. 2 for gender111For the sake of simplicity, we assume two genders throughout this paper\.\)\. However, we noticed that our probes tended to overfit on the speakers in our training set, resulting in huge performance differences in SGID between our train and test sets\. \(This is the manifestation of the effect of the fundamental difference between SGI and SID\!\)\. Our probes were not only learning general SG features, they were learning to map specific speakers to SG classes\. To counteract this, we split our probe into two layers \(see Equation[1](https://arxiv.org/html/2606.10654#S3.E1)\): one projection layerMPM\_\{P\}of small latent dimensionalitydPd\_\{P\}, i\.e\. 5, and one classification layerMCM\_\{C\}, withMP∈ℝdM,dP,MC∈ℝdP,\|S​G​C\|M\_\{P\}\\in\\mathbb\{R\}^\{d\_\{M\},d\_\{P\}\},M\_\{C\}\\in\\mathbb\{R\}^\{d\_\{P\},\|SGC\|\}\. We then learn two linear connections leading from this layer: 1\) the SG classification layer as before, and 2\) a speaker identification layer, which we train using a reverse gradient layer\[[11](https://arxiv.org/html/2606.10654#bib.bib146)\]\. This forces the intermediate layers to be agnostic to speaker identification and prevents speaker overfitting\. We first warm up all three layers until convergence \(without propagating the reverse gradient toMPM\_\{P\}\), then add the gradient reversal and train until a second convergence, as described in\[[32](https://arxiv.org/html/2606.10654#bib.bib8)\]\.

The low\-dimensional, intermediate embeddings from the projection layer contain embeddings from which a SG classifier can be learned; thus, these intermediate embeddings can be viewed as a projection onto theS​G​CSGCspace\. This two\-tiered probe does not add any complexity to our probing procedure, as we use no non\-linear activation function betweenMPM\_\{P\}andMCM\_\{C\}222We experimented with two\-layer MLPs with ReLU activation and found no significant improvement in SGI detection\.

MS​G​C​\(u\):=MC​MP​\(u\)\\displaystyle\\text\{M\}\_\{SGC\}\(u\):=M\_\{C\}M\_\{P\}\(u\)\\quad\(∈ℝ\|S​G​C\|\)\\displaystyle\(\\in\\mathbb\{R\}^\{\|SGC\|\}\)\(1\)MSID,S​G​C​\(u1,u2\):=MS​I​D​MP​\(u\)\\displaystyle\\text\{M\}\_\{\\text\{SID\},SGC\}\(u\_\{1\},u\_\{2\}\):=M\_\{SID\}M\_\{P\}\(u\)\\quad\(∈ℝ\|speakers\|\)\\displaystyle\(\\in\\mathbb\{R\}^\{\|\\text\{speakers\}\|\}\)\(2\)
In the experiments presented in this paper, we useddP=5d\_\{P\}=5, though we experimented with5∈\[5,10,50\]5\\in\[5,10,50\]and found similar results\.

### 3\.1Self\-supervised speech encoder models \(S3Ms\)

We analyzed the capacity of two main S3M architectures,Wav2Vec 2\.0\(W2V2\) andWavLM\. We chose completely open\-source checkpoints of these models, which we can therefore compare according to their architecture and training data\. We usedW2V2as it was the first to use the transformer architecture in S3M speech modeling and is the most widely used today\[[2](https://arxiv.org/html/2606.10654#bib.bib135),[28](https://arxiv.org/html/2606.10654#bib.bib20)\]\. Furthermore, the open\-weight accessibility of many configurations ofW2V2on Hugging Face greatly facilitated our analysis\[[2](https://arxiv.org/html/2606.10654#bib.bib135)\]\. We also analyzedWavLM\. Not only is it by far the best open\-source performer on the SUPERB benchmark; its pretraining lends it to be a good a priori candidate for encoding SGI\. First, during pretraining it uses a reconstruction loss rather than the contrastive loss ofW2V2, as inspired by HuBERT\[[12](https://arxiv.org/html/2606.10654#bib.bib67)\]\. This requirement to be able to reproduce fundamental information about the initial signal might makeWavLMmore likely to store SGI\. \(Previous work has shown pretraining objective to play an important role in the information captured by S3Ms\[[5](https://arxiv.org/html/2606.10654#bib.bib115)\]\)\. Secondly,WavLMuses an additional utterance mixing step in pretraining which is meant to enhance embeddings with speaker\-specific information\[[3](https://arxiv.org/html/2606.10654#bib.bib137)\]\.WavLMpretraining involves randomly overlaying unrelated audio during to simulate a noisy speaking environment, so the model must learn how to differentiate between different audio sources and thus might learn to better model SGs\.

We perform most of our experiments on thelargeconfiguration of S3Ms\. This allows for standardization of model size \(24 hidden layers, hidden layer size 1024,∼300​m\\sim 300mtrainable parameters\)\. We also experimented with thebaseconfiguration ofWavLM\(∼100​m\\sim 100mparameters\)\. This will allow us to ascertain whetherlargemodel size is prerequisite for modeling SGI\.

For maximum comparability, we focused on models trained on a version of LibriVox \(LibriLight, VoxPopuli\), which is the case for theWavLMmodels andW2V2\-lv60\. However, we also compare the 53\-languageW2V2\-XLSR\-53\. This will allow us to evaluate the importance of diverse pretraining data in some SGID tasks\.

### 3\.2Finetuning

Along with model architecture, we also varied our experiments between models that were eitherpretrained onlyorpretrained and finetuned\. We investigate whether the process of finetuning on ASR or SID leads models to store SGI or the opposite\. We also experimented with the Domain Adversarial Training \(DAT\) and Domain Enhancing Training \(DET\) finetuning variants, which have been known to improve overall transcription performance\[[32](https://arxiv.org/html/2606.10654#bib.bib8)\]as well as fairness\[[13](https://arxiv.org/html/2606.10654#bib.bib118)\]\. DET training works by appending an additional SG classifier to any given transformer layer \(usually a middle layer\) to force the middle layer to retain speaker information during finetuning\. DAT training works in the opposite manner, by placing an SG classifier on a late layer with a reverse gradient update rule, to force the model to learn speaker invariant representations prior to ASR\. This is the same strategy as we used for training speaker\-invariant probes in the previous section\.

In accordance with\[[32](https://arxiv.org/html/2606.10654#bib.bib8)\]we used speaker ids as classes for our multi\-class objective, with DET classifier on the 10th layer and DAT classifier on the 21st forlargemodels, and 5th and 9th respectively forbase\. For our SID classifiers, we train an xvector\[[25](https://arxiv.org/html/2606.10654#bib.bib140)\]based on the activations of each layer, an architecture that has been shown to capture speaker information efficiently333We also experimented with linear classifiers but found this less effective in erasing SGI in adversarial layers\. This is likely due to speaker identity in embedding space being of greater dimension than one, and thus a single linear adversarial classifier cannot erase it entirely\[[21](https://arxiv.org/html/2606.10654#bib.bib147)\]\. For ASR, we used a three layer MLP and CTC loss\. We used finetuning recipes provided by the SpeechBrain toolkit\[[20](https://arxiv.org/html/2606.10654#bib.bib113)\]\.

We finetuned our models using a subset of theCommonVoice16 English dataset\. We chose this dataset due its large size and diversity of speakers, and availability of speaker IDs\. One potential downside of this choice is that the datasets we use for SGI detection are from smart speaker commands, which is a different type of speech and thus potentially less than perfectly compatible with our finetuned models\.

### 3\.3Speaker group data

To evaluate models’ detection of SGI, we used two datasets designed for researching fairness in ASR\. Both theMeta Fair\-speech\[[27](https://arxiv.org/html/2606.10654#bib.bib86)\]andSonos Voice Control Bias Assessment\[[23](https://arxiv.org/html/2606.10654#bib.bib112)\]datasets are based on recordings of paid amateur participants performing smart speaker commands lasting several seconds\. Each command is annotated with precise speaker metadata\.Sonoscontains 170K utterances and is annotated withgender\(2 classes444Male; Female\),dialect\(8 classes5556 regional dialects within the USA: Inland\-north; Mid\-Atlantic; Midland; New England; Southern; Western\. Also non\-native speakers ofeitheran Asian languageorSpanish\.\), and speakerage\(5 classes\)6669\-16; 17\-28; 29\-41; 42\-54; 55\-100\.\. It also contains speaker IDs \(1038 speakers in aggregate over train, valid, and test sets\) which we used to compare the difficulty of SID vs SGI probing\.Metacontains 26\.5K utterances and is annotated withgender\(2 classes777Male; female\.\),ethnicity\(7 classes888Asian, South Asian or Asian American; Black or African American; Hispanic, Latino or Spanish; Middle Eastern or North African; Native American, American Indian, or Alaska Native; Native Hawaiian or Other Pacific Islander; White\.\),socio\-economic background\(3 classes999Low, Medium, Affluent\), speakerage\(4 classes10101018\-22; 23\-30; 31\-45; 46\-65\), andnative speaker\(2 classes\)\. We used the train\-test split provided bySonosfor training our SGI probes; we created our own split forMetaas at contains none111111Metaalso doesn’t contain speaker IDs; in order to avoid data leakage of having the same speaker in both train and test splits, we created pseudo\-ids based on unique combinations of all speaker attributes\. However, this was not as fine\-grained a split as possible, as we ended up with far fewer pseudo\-ids \(146\) asMetareported as appearing in the corpus \(593\)\.\.

## 4Results

We illustrate the macro F1 probing accuracy for each SGC probe and model in Figure[1](https://arxiv.org/html/2606.10654#S4.F1)\. Every model we tested achieves greater than random performance for most SGCs on both datasets, though the SGs inMetawere less well detected\. Ethnicity is barely detected by some models, and not detected by others\. Socio\-economic background was not consistently detectable by our probes for any models, and is not pictured in Figure[1](https://arxiv.org/html/2606.10654#S4.F1)\.We note the vastly superior performance of the SID task vs any individual SGI probing task\. Despite having over 1000 classes \(vs a handful for the SGI tasks\), our probes achieved almost perfect performance for SID, while they were nowhere near perfect for any SGC besides gender\.

![Refer to caption](https://arxiv.org/html/2606.10654v1/img/by_layer_line_graph_vertical_both_macro_valid_model_ft_typexvector_dat.png)Figure 1:SGI captured by S3Ms, measured usinglinear probes, as described in Equation[1](https://arxiv.org/html/2606.10654#S3.E1)\.### 4\.1Impact of model pretraining on SG encoding

The pretrained model architectures \(black\) which captures the most SGI areWavLM\-large\. This aligns with our hypothesis that its reconstructive loss would lend itself more to SG encoding than a contrastive loss such as employed byW2V2models\. Unsurprisingly,WavLM\-large captures more SGI thanWavLM\-base for most categories\. MultilingualW2V2tended to capture about the same amount of SGI asW2V2pretrained on English only, which suggests that multilingual pretraining by itself does not force the model to learn such features\.

### 4\.2Finetuning for ASR/SID reveals two classes of SGCs

For all model variants depicted in Figure[1](https://arxiv.org/html/2606.10654#S4.F1), we observe that \(apart from gender\) comparatively little SGI is captured in the earliest layers of most models\. This increases until just before halfway, i\.e\. layer 10 forlargeand 5 forbasemodels\. In middle and later layers, pretrained models \(black\) tend to retain relatively consistent levels of SGI for most classes, though the final layers ofW2V2models break this pattern \(behavior which was also observed in\[[18](https://arxiv.org/html/2606.10654#bib.bib124)\]\)\.

However, in the finetuned models, there aretwo main behavior patternswe observe in themiddle and later layers\. Focusing on the models finetuned on vanilla ASR using CTC \(green\), we note some SGCs remain detectable atrelatively consistent levels\- these are dialect, ethnicity, and is\_native\. The detection rate of these SGCs tends to continue increasing past the halfway layer, and subsequently either decreases only slightly or even increases \(for is\_native\), despite model finetuning\. However, other SGCs becomeconsiderably less detectablein later layers \- these are gender and age\. In contrast, consider the models finetuned on the SID task \(blue\)\. SID finetuned models permit detection of gender and age \(as well as speaker\_id\) through later layers to a much greater extent than do the vanilla CTC models, and often greater than the pretrained models as well\. However, SID finetuning does not improve detection of dialect, ethnicity, or is\_native \(though it doesn’t attenuate it either\)\.

We hypothesize that this contrastive behavior is due to nature in which different SGs are manifested vocally\. S3Ms are known to discard information during finetuning not relevant to the downstream task at hand\[[31](https://arxiv.org/html/2606.10654#bib.bib50)\]\. Our observations thus support the notion that some SGCs are useful in ASR, while other SGCs are useful in SID\. First, the SGCs that are useful for ASR are those which exhibit higher interclass semantic variance\. The differences between different dialects’, ethnicities’, and native vs non\-native speakers’ speech tend to be prosodic/semantic \- non\-native speakers might insert pauses where native speakers would not pause; speakers of different dialects might use different phones to represent the same grapheme; speakers of different ethnicities might use different graphemes altogether\[[6](https://arxiv.org/html/2606.10654#bib.bib120),[7](https://arxiv.org/html/2606.10654#bib.bib2)\]\. It is therefore intuitive that these SGCs remain relatively well detectable in later layers\.

On the other hand, the SGCs which are associated with SID finetuning tend to be more phonetically variant\. The primary distinguishing factors between the speech of different age groups and genders is more closely related to phonetic \(women and children tend to have higher pitched voices, older people tend to have lower pitched voices, etc\.\)\. Previous work has shown that phonetic information encoded by S3Ms peaks towards the middle of models and then decreases, whereas semantic information peaks towards later layers\[[18](https://arxiv.org/html/2606.10654#bib.bib124),[19](https://arxiv.org/html/2606.10654#bib.bib125)\]\- it is therefore not surprising that phonetically variant SGC detection peaks earlier for all model variants \(though they retain their plateau with SID finetuning\), while semantically variant SGC detection peaks slightly later\.

### 4\.3Ramifications of fairer ASR finetuning on SGI detection

SGI detection on models finetuned with fairness enhancing ASR algorithms, i\.e\. CTC \+ DET \(crimson\) and CTC \+ DAT \(orange\) followed a similar pattern as we established in the previous section\. Phonetic SGC was better detectable in CTC \+ DET than for vanilla CTC\. On the other hand, models finetuned with CTC \+ DAT tend to slough off even more phonetically variant SGI in later layers than do vanilla CTC models \- again this follows our intuition\. However, CTC \+ DET models did not tend to encode semantically variant SGCs any better than vanilla ASR, nor did CTC \+ DAT models lose much by way of semantically variant SGI in later layers\. This supports the notion that SID\-based multi\-task ASR finetuning should have the strongest effect on forcing performance invariance between phonetically variant SGCs, while it should be less effective on equalizing the performance between semantically variant SGCs\. However, it is precisely the semantically variant classes that exhibit the greatest divergence in fairness for which DET and DAT finetuning appear to be least helpful\[[23](https://arxiv.org/html/2606.10654#bib.bib112)\]\. This motivates further study into alternative DET and DAT methods, such as were proposed in\[[13](https://arxiv.org/html/2606.10654#bib.bib118)\], that use accent labels as targets, perhaps in addition to speaker IDs\.

![Refer to caption](https://arxiv.org/html/2606.10654v1/img/cos_sim_centroids10.png)Figure 2:Cosine similarity between principal components of each SGC compared with principal components of the speaker centroid matrix, for layer 10 \(i\.e\. layer with DET head\) ofWavLM\-large finetuned on CTC \+ DET \+ DAT\.![Refer to caption](https://arxiv.org/html/2606.10654v1/img/cos_sim_centroids21.png)Figure 3:Cosine similarity between principal components of each SGC compared with principal components of the speaker centroid matrix, for layer 21 \(i\.e\. layer with DAT head\) ofWavLM\-large finetuned on CTC \+ DET \+ DAT\.

## 5Ablation study on specific embedding dimensions

### 5\.1SGI is distributed across frames

In Figure[4](https://arxiv.org/html/2606.10654#S5.F4), we observe the effect of pooling over different subsequences of the latent embeddings\. We first note, not surprisingly, that pooling over 50 frames \(long dashes\) allows for far better SGI detection than single frames \(short dashes\)\. Previous work in\[[9](https://arxiv.org/html/2606.10654#bib.bib105)\]showed that the final frames of an encoded utterance contain most speaker ID information than earlier frames; however, we observe the opposite of this\. For some SGCs \(like gender\), the difference is mostly negligible in comparing the pooled first second of audio and the final second of audio\. However, for all other SGCs \(which are more difficult to detect\), pooling over the first 50 frames led to much greater SGI detection than the final 50 frames, a result more consistent with\[[17](https://arxiv.org/html/2606.10654#bib.bib95)\]\(though their analysis focused on word embeddings\)\. The significantly superior performance of earlier layers could in part be due to the smart\-speaker command format of our data \- the regularity of the wake phrase at the start of each audio might provide a stable basis for SGI encoding\.

![Refer to caption](https://arxiv.org/html/2606.10654v1/img/by_layer_line_graph_vertical_both_macro_valid_embedding_type.png)Figure 4:Earlier frames of a pretrained S3M capture more SGI than later frames, though the entire frame captures most of all\.
### 5\.2Different SGCs are encoded across different latent dimensions

\[[15](https://arxiv.org/html/2606.10654#bib.bib93)\]show that SID information and phonetic information are stored along orthogonal dimensions within latent embeddings of S3Ms\. They show this by comparing the principal components of the centroid matrix for each class, in their case phonemes vs speaker IDs\. The centroid matrix is composed of the centroid \(i\.e\. the mean over all samples\) for each class \(i\.e\. for gender that would mean two classes\)\. The principal components of this matrix \(measured by PCA\) are the directions along which there is maximum variance between centroids of that SGC\. If principal components of centroid matricesMS​G​C1,MS​G​C2M\_\{SGC\_\{1\}\},M\_\{SGC\_\{2\}\}are orthogonal \(i\.e\. cosine similarity close to 0\), that implies that the information encapsulated byS​G​C1SGC\_\{1\}andS​G​C2SGC\_\{2\}are stored along orthogonal dimensions\.

In Figures[2](https://arxiv.org/html/2606.10654#S4.F2)and[3](https://arxiv.org/html/2606.10654#S4.F3), we depict the cosine similarity \(SCS\_\{C\}\) between pairs of principal components of SGCs\. Higher values indicate correlated principal components\. Particularly noteworthy are the values in the top left corner of each plot \- these correspond to the covariance of the greatest principal components of SID with each SGC\. Note the high degree of correlation between principal components for all SGCs at layer 10 \(the DET layer\); however, in layer 21 \(the DAT layer\), is\_native and and gender \(forSonos\) correlate less with the first principal components of speaker\_id\. This implies that DAT forces gender and is\_nativeSonosto be modeled in an orthogonal dimension to speaker\_id, rather than render the layer 21 embeddings invariant to those SGCs121212In Figure[1](https://arxiv.org/html/2606.10654#S4.F1)we saw that CTC \+ DAT finetuning hampered the model’s ability to detect speakers’ gender\. It is therefore puzzling thatSonosgender and speaker\_id are so uncorrelated\.

## 6Outlook

In this paper we provide a framework through which to evaluate S3Ms’ ability to detect SGs\. We show that a model’s pretraining has a strong effect on how much SGI it captures\. We also show that forcing an ASR\-finetuned S3M to be SID invariant in its final layers renders the model blind towards more phonetically variant SGCs like gender and age; this forms a theoretical foundation on why DAT works for phonetic SGCs\. However, semantically variant SGI \(such as dialect or is\_native\) tends to be retained in later layers in pretrained and finetuned models; a DAT head forcing speaker invariance has negligible effect on semantic SGI encoding\. In our ablation study we showed that forcing speaker invariance causes certain SGCs to be encoded on orthogonal axes to speaker\_id\. This further supports the hypothesis that DAT is less than optimally effective in suppressing those SGCs in later layers\. That said, the extent to which SGI is necessary for high downstream task performance demands further study\. Based on our results, we recommend experiment with variants of DET/DAT combining multiple SGC classifiers to amplify specific types of SGI in specific model layers\.

## References

- \[1\]\(2022\-01\)Similarities between Arabic dialects: Investigating geographical proximity\.Information Processing & Management59\(1\),pp\. 102770\.External Links:ISSN 0306\-4573,[Document](https://dx.doi.org/10.1016/j.ipm.2021.102770)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p3.1)\.
- \[2\]A\. Baevski, H\. Zhou, A\. Mohamed, and M\. Auli\(2020\-10\)Wav2vec 2\.0: A Framework for Self\-Supervised Learning of Speech Representations\.arXiv\.External Links:2006\.11477Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.10654#S3.SS1.p1.1)\.
- \[3\]S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao, J\. Wu, L\. Zhou, S\. Ren, Y\. Qian, Y\. Qian, J\. Wu, M\. Zeng, X\. Yu, and F\. Wei\(2022\-10\)WavLM: Large\-Scale Self\-Supervised Pre\-Training for Full Stack Speech Processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.External Links:2110\.13900,ISSN 1932\-4553, 1941\-0484,[Document](https://dx.doi.org/10.1109/JSTSP.2022.3188113)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.10654#S3.SS1.p1.1)\.
- \[4\]K\. Choi, A\. Pasad, T\. Nakamura, S\. Fukayama, K\. Livescu, and S\. Watanabe\(2024\-06\)Self\-Supervised Speech Representations are More Phonetic than Semantic\.arXiv\.External Links:2406\.08619Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1)\.
- \[5\]Y\. Chung, Y\. Belinkov, and J\. Glass\(2021\-02\)Similarity Analysis of Self\-Supervised Speech Representations\.arXiv\.External Links:2010\.11481,[Document](https://dx.doi.org/10.48550/arXiv.2010.11481)Cited by:[§3\.1](https://arxiv.org/html/2606.10654#S3.SS1.p1.1)\.
- \[6\]S\. Coulange and S\. Rossato\(2020\)Proximité rythmique entre apprenants et natifs du français Évaluation d’une métrique basée sur le CEFC\.InActes de La 6e Conférence Conjointe Journées d’Études Sur La Parole \(JEP, 33e Édition\), Traitement Automatique Des Langues Naturelles \(TALN, 27e Édition\), Rencontre Des Étudiants Chercheurs En Informatique Pour Le Traitement Automatique Des Langues \(RÉCITAL, 22e Édition\)\. Volume 1 : Journées d’Études Sur La Parole,C\. Benzitoun, C\. Braud, L\. Huber, D\. Langlois, S\. Ouni, S\. Pogodalla, and S\. Schneider \(Eds\.\),Nancy, France,pp\. 118–126\.Cited by:[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p3.1)\.
- \[7\]R\. Dorn\(2019\-09\)Dialect\-Specific Models for Automatic Speech Recognition of African American Vernacular English\.InProceedings of the Student Research Workshop Associated with RANLP 2019,V\. Kovatchev, I\. Temnikova, B\. Šandrih, and I\. Nikolova \(Eds\.\),Varna, Bulgaria,pp\. 16–20\.External Links:[Document](https://dx.doi.org/10.26615/issn.2603-2821.2019%5F003)Cited by:[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p3.1)\.
- \[8\]S\. Evain, H\. Nguyen, H\. Le, M\. Z\. Boito, S\. Mdhaffar, S\. Alisamir, Z\. Tong, N\. Tomashenko, M\. Dinarelli, T\. Parcollet, A\. Allauzen, Y\. Esteve, B\. Lecouteux, F\. Portet, S\. Rossato, F\. Ringeval, D\. Schwab, and L\. Besacier\(2021\-08\)LeBenchmark: A Reproducible Framework for Assessing Self\-Supervised Representation Learning from Speech\.InInterspeech 2021,pp\. 1439–1443\.External Links:2104\.11462,[Document](https://dx.doi.org/10.21437/Interspeech.2021-556)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1)\.
- \[9\]C\. Feng, P\. Hsu, and H\. Lee\(2022\-05\)Silence is Sweeter Than Speech: Self\-Supervised Model Using Silence to Store Speaker Information\.arXiv\.External Links:2205\.03759,[Document](https://dx.doi.org/10.48550/arXiv.2205.03759)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p2.1),[§3](https://arxiv.org/html/2606.10654#S3.p1.1),[§5\.1](https://arxiv.org/html/2606.10654#S5.SS1.p1.1)\.
- \[10\]S\. Feng, B\. M\. Halpern, O\. Kudina, and O\. Scharenborg\(2024\-03\)Towards inclusive automatic speech recognition\.Computer Speech & Language84,pp\. 101567\.External Links:ISSN 0885\-2308,[Document](https://dx.doi.org/10.1016/j.csl.2023.101567)Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p2.1)\.
- \[11\]Y\. Ganin, E\. Ustinova, H\. Ajakan, P\. Germain, H\. Larochelle, F\. Laviolette, M\. Marchand, and V\. Lempitsky\(2017\)Domain\-Adversarial Training of Neural Networks\.InDomain Adaptation in Computer Vision Applications,G\. Csurka \(Ed\.\),pp\. 189–209\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-58347-1%5F10),ISBN 978\-3\-319\-58346\-4 978\-3\-319\-58347\-1Cited by:[§3](https://arxiv.org/html/2606.10654#S3.p2.8)\.
- \[12\]W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed\(2021\-06\)HuBERT: Self\-Supervised Speech Representation Learning by Masked Prediction of Hidden Units\.arXiv\.External Links:2106\.07447Cited by:[§3\.1](https://arxiv.org/html/2606.10654#S3.SS1.p1.1)\.
- \[13\]J\. Li, V\. Manohar, P\. Chitkara, A\. Tjandra, M\. Picheny, F\. Zhang, X\. Zhang, and Y\. Saraf\(2021\-10\)Accent\-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings\.arXiv\.External Links:2110\.03520,[Document](https://dx.doi.org/10.48550/arXiv.2110.03520)Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p2.1),[§3\.2](https://arxiv.org/html/2606.10654#S3.SS2.p1.1),[§4\.3](https://arxiv.org/html/2606.10654#S4.SS3.p1.1)\.
- \[14\]L\. Lima, V\. Furtado, E\. Furtado, and V\. Almeida\(2019\-05\)Empirical Analysis of Bias in Voice\-based Personal Assistants\.InCompanion Proceedings of The 2019 World Wide Web Conference,San Francisco USA,pp\. 533–538\.External Links:[Document](https://dx.doi.org/10.1145/3308560.3317597),ISBN 978\-1\-4503\-6675\-5Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p2.1)\.
- \[15\]O\. Liu, H\. Tang, and S\. Goldwater\(2023\-12\)Self\-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces\.arXiv\.External Links:2305\.12464,[Document](https://dx.doi.org/10.48550/arXiv.2305.12464)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2606.10654#S5.SS2.p1.3)\.
- \[16\]M\. Mohamed, O\. D\. Liu, H\. Tang, and S\. Goldwater\(2024\-06\)Orthogonality and isotropy of speaker and phonetic information in self\-supervised speech representations\.arXiv\.External Links:2406\.09200,[Document](https://dx.doi.org/10.48550/arXiv.2406.09200)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p2.1)\.
- \[17\]A\. Pasad, C\. Chien, S\. Settle, and K\. Livescu\(2024\-01\)What Do Self\-Supervised Speech Models Know About Words?\.arXiv\.External Links:2307\.00162,[Document](https://dx.doi.org/10.48550/arXiv.2307.00162)Cited by:[§5\.1](https://arxiv.org/html/2606.10654#S5.SS1.p1.1)\.
- \[18\]A\. Pasad, J\. Chou, and K\. Livescu\(2022\-12\)Layer\-wise Analysis of a Self\-supervised Speech Representation Model\.arXiv\.External Links:2107\.04734Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p4.1)\.
- \[19\]A\. Pasad, B\. Shi, and K\. Livescu\(2023\-03\)Comparative layer\-wise analysis of self\-supervised speech models\.arXiv\.External Links:2211\.03929Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p4.1)\.
- \[20\]M\. Ravanelli, T\. Parcollet, P\. Plantinga, A\. Rouhe, S\. Cornell, L\. Lugosch, C\. Subakan, N\. Dawalatabad, A\. Heba, J\. Zhong, J\. Chou, S\. Yeh, S\. Fu, C\. Liao, E\. Rastorgueva, F\. Grondin, W\. Aris, H\. Na, Y\. Gao, R\. D\. Mori, and Y\. Bengio\(2021\-06\)SpeechBrain: A General\-Purpose Speech Toolkit\.arXiv\.External Links:2106\.04624,[Document](https://dx.doi.org/10.48550/arXiv.2106.04624)Cited by:[§3\.2](https://arxiv.org/html/2606.10654#S3.SS2.p2.1)\.
- \[21\]S\. Ravfogel, M\. Twiton, Y\. Goldberg, and R\. Cotterell\(2024\-12\)Linear Adversarial Concept Erasure\.arXiv\.External Links:2201\.12091,[Document](https://dx.doi.org/10.48550/arXiv.2201.12091)Cited by:[footnote 3](https://arxiv.org/html/2606.10654#footnote3)\.
- \[22\]R\. Sanabria, H\. Tang, and S\. Goldwater\(2023\-03\)Analyzing Acoustic Word Embeddings from Pre\-trained Self\-supervised Speech Models\.arXiv\.External Links:2210\.16043,[Document](https://dx.doi.org/10.48550/arXiv.2210.16043)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1)\.
- \[23\]C\. Sekkat, F\. Leroy, S\. Mdhaffar, B\. P\. Smith, Y\. Estève, J\. Dureau, and A\. Coucke\(2024\-05\)Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 15056–15075\.Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p2.1),[§3\.3](https://arxiv.org/html/2606.10654#S3.SS3.p1.1),[§4\.3](https://arxiv.org/html/2606.10654#S4.SS3.p1.1)\.
- \[24\]S\. Shon, A\. Ali, Y\. Samih, H\. Mubarak, and J\. Glass\(2020\-05\)ADI17: A Fine\-Grained Arabic Dialect Identification Dataset\.InICASSP 2020 \- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Barcelona, Spain,pp\. 8244–8248\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP40776.2020.9052982),ISBN 978\-1\-5090\-6631\-5Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p3.1)\.
- \[25\]D\. Snyder, D\. Garcia\-Romero, G\. Sell, D\. Povey, and S\. Khudanpur\(2018\-04\)X\-Vectors: Robust DNN Embeddings for Speaker Recognition\.In2018 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 5329–5333\.External Links:ISSN 2379\-190X,[Document](https://dx.doi.org/10.1109/ICASSP.2018.8461375)Cited by:[§3\.2](https://arxiv.org/html/2606.10654#S3.SS2.p2.1)\.
- \[26\]B\. van Niekerk, L\. Nortje, M\. Baas, and H\. Kamper\(2021\-08\)Analyzing Speaker Information in Self\-Supervised Models to Improve Zero\-Resource Speech Processing\.arXiv\.External Links:2108\.00917,[Document](https://dx.doi.org/10.48550/arXiv.2108.00917)Cited by:[§3](https://arxiv.org/html/2606.10654#S3.p1.1)\.
- \[27\]I\. Veliche, Z\. Huang, V\. A\. Kochaniyan, F\. Peng, O\. Kalinli, and M\. L\. Seltzer\(2024\-08\)Towards measuring fairness in speech recognition: Fair\-Speech dataset\.arXiv\.External Links:2408\.12734,[Document](https://dx.doi.org/10.48550/arXiv.2408.12734)Cited by:[§3\.3](https://arxiv.org/html/2606.10654#S3.SS3.p1.1)\.
- \[28\]R\. Whetten, T\. Parcollet, M\. Dinarelli, and Y\. Estève\(2024\-09\)Open Implementation and Study of BEST\-RQ for Speech Processing\.arXiv\.External Links:2405\.04296,[Document](https://dx.doi.org/10.48550/arXiv.2405.04296)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.10654#S3.SS1.p1.1)\.
- \[29\]Y\. Wu, D\. Rough, A\. Bleakley, J\. Edwards, O\. Cooney, P\. R\. Doyle, L\. Clark, and B\. R\. Cowan\(2020\-10\)See What I’m Saying? Comparing Intelligent Personal Assistant Use for Native and Non\-Native Language Speakers\.In22nd International Conference on Human\-Computer Interaction with Mobile Devices and Services,MobileHCI ’20,New York, NY, USA,pp\. 1–9\.External Links:[Document](https://dx.doi.org/10.1145/3379503.3403563),ISBN 978\-1\-4503\-7516\-0Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p2.1)\.
- \[30\]S\. Yang, P\. Chi, Y\. Chuang, Cheng\-I\. J\. Lai, K\. Lakhotia, Y\. Y\. Lin, A\. T\. Liu, J\. Shi, X\. Chang, G\. Lin, T\. Huang, W\. Tseng, K\. Lee, D\. Liu, Z\. Huang, S\. Dong, S\. Li, S\. Watanabe, A\. Mohamed, and H\. Lee\(2021\-10\)SUPERB: Speech processing Universal PERformance Benchmark\.arXiv\.External Links:2105\.01051,[Document](https://dx.doi.org/10.48550/arXiv.2105.01051)Cited by:[§1](https://arxiv.org/html/2606.10654#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p1.1),[§3](https://arxiv.org/html/2606.10654#S3.p2.8)\.
- \[31\]S\. Zaiem, T\. Parcollet, and S\. Essid\(2024\-06\)Less Forgetting for Better Generalization: Exploring Continual\-learning Fine\-tuning Methods for Speech Self\-supervised Representations\.arXiv\.External Links:2407\.00756,[Document](https://dx.doi.org/10.48550/arXiv.2407.00756)Cited by:[§4\.2](https://arxiv.org/html/2606.10654#S4.SS2.p3.1)\.
- \[32\]W\. Zhou, H\. Wu, J\. Xu, M\. Zeineldeen, C\. Lüscher, R\. Schlüter, and H\. Ney\(2023\-06\)Enhancing and Adversarial: Improve ASR with Speaker Labels\.InICASSP 2023 \- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.External Links:2211\.06369,[Document](https://dx.doi.org/10.1109/ICASSP49357.2023.10096722)Cited by:[§2\.1](https://arxiv.org/html/2606.10654#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2606.10654#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2606.10654#S3.SS2.p2.1),[§3](https://arxiv.org/html/2606.10654#S3.p2.8)\.