Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders

# DISENTANGLING LINGUISTIC AND PARALINGUISTIC INFORMATION WITH ROUTED SPARSE AUTOENCODERS
Source: [https://arxiv.org/html/2610.10865](https://arxiv.org/html/2610.10865)
###### Abstract

Self\-supervised speech encoders contain linguistic and paralinguistic information in a shared, entangled representation space\. We combine a TopK sparse autoencoder with route\-specific supervision and cross\-factor adversaries\. Across frozen SPEAR and WavLM encoders, independent probes show factor\-specific retention and suppression: linguistic information remains stronger in the linguistic route, while paralinguistic factors, including speaker identity, emotion, and prosody, are retained in the paralinguistic route and substantially reduced in the linguistic route\. The route organisation learned on LibriSpeech persists on MSP\-Podcast without representation\-side retraining\. Feature\-space route interventions further transfer the swapped factor while largely preserving the information carried by the unchanged route\. These results show consistent route\-selective separation across encoders, corpora, independent probes, and representation\-level interventions\.

###### Index Terms:

Sparse autoencoders, speech representation learning, disentanglement

††address:Department of Engineering, University of Cambridge, Cambridge, UK## 1Introduction

Self\-supervised learning \(SSL\) has become central to speech representation learning, enabling speech encoders to learn from large amounts of unlabelled audio and produce representations that support tasks including phone recognition, automatic speech recognition, speaker identification, emotion recognition, and prosody\-related prediction\[[2](https://arxiv.org/html/2610.10865#bib.bib1),[9](https://arxiv.org/html/2610.10865#bib.bib5),[5](https://arxiv.org/html/2610.10865#bib.bib2),[22](https://arxiv.org/html/2610.10865#bib.bib12)\]\. However, these different properties exist in a shared and entangled representation space\. The same activation can carry phonetic content, speaker identity, prosody, and other information, making selective interpretation, removal, or modification of individual properties difficult\. Hence, separating linguistic and paralinguistic information would support clearer interpretation and more selective control of speech representations\.

Speech representation disentanglement has been addressed using perturbation\-based invariance, information bottlenecks, architectural mechanisms, and adversarial training\[[19](https://arxiv.org/html/2610.10865#bib.bib9),[18](https://arxiv.org/html/2610.10865#bib.bib17),[12](https://arxiv.org/html/2610.10865#bib.bib6),[7](https://arxiv.org/html/2610.10865#bib.bib3),[21](https://arxiv.org/html/2610.10865#bib.bib10)\]\. Sparse autoencoders \(SAEs\) provide a complementary approach by mapping a dense representation into a wider sparse code and reconstructing the original representation from a small subset of active units\. Previous work has identified interpretable units in language, speech, and audio representations\[[3](https://arxiv.org/html/2610.10865#bib.bib16),[16](https://arxiv.org/html/2610.10865#bib.bib7),[14](https://arxiv.org/html/2610.10865#bib.bib20),[1](https://arxiv.org/html/2610.10865#bib.bib21)\]\. However, sparsity alone does not ensure that the learned units correspond to factors selected in advance\. SAE dictionaries can depend on model width and training choices, and a reconstruction objective need not recover a canonical or predefined factorisation\[[11](https://arxiv.org/html/2610.10865#bib.bib13),[17](https://arxiv.org/html/2610.10865#bib.bib8)\]\. Existing speech and audio SAE studies primarily analyse discovered features rather than routing the sparse code into task\-specific subspaces under an explicit disentanglement objective, which is the gap addressed here\.

In this work, we investigate whether the sparse code generated by an SAE on top of an SSL speech encoder can be organised into linguistic and paralinguistic routes with factor\-specific retention and suppression\. The linguistic route retains phonetic and transcript information, while the paralinguistic route retains speaker identity, emotion, and prosody\. Phone recognition \(PR\) provides the linguistic supervision signal, while speaker identification \(SID\), emotion recognition \(ER\), and prosody prediction provide the paralinguistic signals\. The proposed disentanglement framework combines a TopK sparse autoencoder\[[13](https://arxiv.org/html/2610.10865#bib.bib14),[8](https://arxiv.org/html/2610.10865#bib.bib4)\]with unit\-level routing, route\-specific task supervision, and cross\-factor adversarial objectives\. The TopK SAE provides an overcomplete sparse representation, while unit\-level routing, task supervision, and adversarial objectives organise its units into linguistic and paralinguistic routes\.

We evaluate the framework using frozen SPEAR\[[23](https://arxiv.org/html/2610.10865#bib.bib11)\]and WavLM\[[5](https://arxiv.org/html/2610.10865#bib.bib2)\]representations\[[15](https://arxiv.org/html/2610.10865#bib.bib18),[23](https://arxiv.org/html/2610.10865#bib.bib11),[5](https://arxiv.org/html/2610.10865#bib.bib2)\]\. Across LibriSpeech and MSP\-Podcast, independent probes evaluate linguistic information through PR and paralinguistic information through SID, ER, and prosody prediction\. The evaluation includes reconstruction\-only, fixed\- routing, and quota\-freeze SAEs, together with Eta\-WavLM\[[20](https://arxiv.org/html/2610.10865#bib.bib23)\], adversarial ablations, frozen cross\-corpus transfer, and controlled route interventions\.

## 2Methodology

We start with a frozen SSL speech encoder that acts as a source of the input representation\. Its representation is fed into a TopK SAE, whose active units are partitioned into linguistic and paralinguistic routes\. The SAE, routing masks, and task heads are trained jointly under reconstruction and task supervision losses, while the speech encoder is frozen\.

![Refer to caption](https://arxiv.org/html/2610.10865v1/Framework_no_z_u_Updated_clean_v3.png)Figure 1:TopK SAE produce𝐳t\\mathbf\{z\}\_\{t\}on top of a frozen speech encoder, which is routed into linguistic𝐳L\\mathbf\{z\}\_\{L\}and paralinguistic𝐳P\\mathbf\{z\}\_\{P\}representations\.### 2\.1Sparse Autoencoder

Given an input waveform, the frozen speech encoder produces frame\-level representations𝐗=𝐱1:T\\mathbf\{X\}=\\mathbf\{x\}\_\{1:T\}, whereTTis the sequence length and𝐱t∈ℝD\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{D\}\. A linear SAE projects this representation into an overcomplete space while enforcing sparsity by selecting the TopK elements\[[13](https://arxiv.org/html/2610.10865#bib.bib14),[8](https://arxiv.org/html/2610.10865#bib.bib4)\]\. The encoder and decoder are defined as

𝐬t\\displaystyle\\mathbf\{s\}\_\{t\}=𝐖enc​𝐱t\+𝐛enc,\\displaystyle=\\mathbf\{W\}\_\{\\mathrm\{enc\}\}\\mathbf\{x\}\_\{t\}\+\\mathbf\{b\}\_\{\\mathrm\{enc\}\},\(1\)𝐳t\\displaystyle\\mathbf\{z\}\_\{t\}=TopK⁡\(𝐬t,k\),\\displaystyle=\\operatorname\{TopK\}\(\\mathbf\{s\}\_\{t\};k\),\(2\)𝐱^t\\displaystyle\\hat\{\\mathbf\{x\}\}\_\{t\}=𝐖dec​𝐳t\+𝐛dec,\\displaystyle=\\mathbf\{W\}\_\{\\mathrm\{dec\}\}\\mathbf\{z\}\_\{t\}\+\\mathbf\{b\}\_\{\\mathrm\{dec\}\},\(3\)where𝐖enc∈ℝK×D\\mathbf\{W\}\_\{\\mathrm\{enc\}\}\\in\\mathbb\{R\}^\{K\\times D\},𝐖dec∈ℝD×K\\mathbf\{W\}\_\{\\mathrm\{dec\}\}\\in\\mathbb\{R\}^\{D\\times K\},K\>DK\>D, andkkis the active\-unit budget\. The reconstruction objective is

ℒrecon=𝔼t​‖𝐱t−𝐱^t‖22\.\\mathcal\{L\}\_\{\\mathrm\{recon\}\}=\\mathbb\{E\}\_\{t\}\\left\\\|\\mathbf\{x\}\_\{t\}\-\\hat\{\\mathbf\{x\}\}\_\{t\}\\right\\\|\_\{2\}^\{2\}\.\(4\)TopK provides exact control over the number of active units without introducing shrinkage on the retained activations\. To prevent dead activations, AuxK uses inactive units to predict the reconstruction residual\[[8](https://arxiv.org/html/2610.10865#bib.bib4)\]:

ℒAuxK=𝔼t​‖\(𝐱t−𝐱^t\)−𝐖dec\(aux\)​𝐳t\(aux\)‖22\.\\mathcal\{L\}\_\{\\mathrm\{AuxK\}\}=\\mathbb\{E\}\_\{t\}\\left\\\|\(\\mathbf\{x\}\_\{t\}\-\\hat\{\\mathbf\{x\}\}\_\{t\}\)\-\\mathbf\{W\}^\{\(\\mathrm\{aux\}\)\}\_\{\\mathrm\{dec\}\}\\mathbf\{z\}^\{\(\\mathrm\{aux\}\)\}\_\{t\}\\right\\\|\_\{2\}^\{2\}\.\(5\)Here,𝐖dec\(aux\)\\mathbf\{W\}^\{\(\\mathrm\{aux\}\)\}\_\{\\mathrm\{dec\}\}and𝐳t\(aux\)\\mathbf\{z\}^\{\(\\mathrm\{aux\}\)\}\_\{t\}are restricted to the selected inactive units\. This encourages them to acquire useful structure and re\-enter the main TopK selection\.

### 2\.2Task\-Specific Routing

Without supervision, SAE units might not align with the factors of interest\[[11](https://arxiv.org/html/2610.10865#bib.bib13),[17](https://arxiv.org/html/2610.10865#bib.bib8)\]\. Each latent unit is therefore mapped to either the linguistic route or the paralinguistic route with learnable masks𝐦L\\mathbf\{m\}\_\{L\}and𝐦P\\mathbf\{m\}\_\{P\}over the latent space:

𝐳L\\displaystyle\\mathbf\{z\}\_\{L\}=𝐦L⊙𝐳t,\\displaystyle=\\mathbf\{m\}\_\{L\}\\odot\\mathbf\{z\}\_\{t\},𝐳P\\displaystyle\\mathbf\{z\}\_\{P\}=𝐦P⊙𝐳t,\\displaystyle=\\mathbf\{m\}\_\{P\}\\odot\\mathbf\{z\}\_\{t\},\(6\)mL​\[u\]\+mP​\[u\]\\displaystyle m\_\{L\}\[u\]\+m\_\{P\}\[u\]=1,\\displaystyle=1,u∈\{1,…,K\}\.\\displaystyle u\\in\\\{1,\\ldots,K\\\}\.\(7\)
The linguistic route𝐳L\\mathbf\{z\}\_\{L\}is trained to retain linguistic information while reducing paralinguistic factor recoverability and the paralinguistic route𝐳P\\mathbf\{z\}\_\{P\}is trained to retain paralinguistic information while reducing linguistic recoverability\. For learned routing, the masks are global vectors shared across frames and utterances\. Each unituuhas trainable routing logits𝐚u\\mathbf\{a\}\_\{u\}, and its assignment is obtained using straight\-through hard Gumbel–Softmax\[[10](https://arxiv.org/html/2610.10865#bib.bib22)\]:

\[mL​\[u\],mP​\[u\]\]=STGumbelSoftmax⁡\(𝐚u;τ\)\.\[m\_\{L\}\[u\],m\_\{P\}\[u\]\]=\\operatorname\{STGumbelSoftmax\}\(\\mathbf\{a\}\_\{u\};\\tau\)\.\(8\)Writingpu,ℓ=softmax⁡\(𝐚u\)ℓp\_\{u,\\ell\}=\\operatorname\{softmax\}\(\\mathbf\{a\}\_\{u\}\)\_\{\\ell\}, the routing loss encourages a decisive assignment:

ℒrouting=−1K∑u=1K∑ℓ∈\{L,P\}pu,ℓlogpu,ℓ\.\\mathcal\{L\}\_\{\\mathrm\{routing\}\}=\-\\frac\{1\}\{K\}\\sum\_\{u=1\}^\{K\}\\sum\_\{\\ell\\in\\\{L,P\\\}\}p\_\{u,\\ell\}\\log p\_\{u,\\ell\}\.\(9\)Fixed routingpredetermines unit membership and applies separate budgetskLk\_\{L\}andkPk\_\{P\}, wherekL\+kP=kk\_\{L\}\+k\_\{P\}=k\. Both membership and budgets remain fixed throughout training\.Quota\-freeze routingfirst learns global assignments while a global TopK selectskkactivations per frame\. At the freeze step, the assignments are fixed, and their mean activity is converted into separate route quotaskLk\_\{L\}andkPk\_\{P\}\. Training then continues with separate TopK selection within each route\.

### 2\.3Task Supervision and Adversarial Training

Task heads attached to each route create supervision signals that align it with its intended factors\. Phone recognition supervises𝐳L\\mathbf\{z\}\_\{L\}, while speaker identification, emotion classification, and prosody prediction supervise𝐳p\\mathbf\{z\}\_\{p\}\.

ℒPR\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{PR\}\}=CTC⁡\(hPR​\(𝐳L\),yph\),\\displaystyle=\\operatorname\{CTC\}\\left\(h\_\{\\mathrm\{PR\}\}\(\\mathbf\{z\}\_\{L\}\),y\_\{\\mathrm\{ph\}\}\\right\),\(10\)ℒSID\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SID\}\}=−log⁡pϕ​\(yspk∣pool⁡\(𝐳P\)\),\\displaystyle=\-\\log p\_\{\\phi\}\\left\(y\_\{\\mathrm\{spk\}\}\\mid\\operatorname\{pool\}\(\\mathbf\{z\}\_\{P\}\)\\right\),\(11\)ℒPros\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Pros\}\}=ProsMSE⁡\(hPros​\(𝐳P\),yPros\),\\displaystyle=\\operatorname\{ProsMSE\}\\left\(h\_\{\\mathrm\{Pros\}\}\(\\mathbf\{z\}\_\{P\}\),y\_\{\\mathrm\{Pros\}\}\\right\),\(12\)ℒEmo\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Emo\}\}=−we​log⁡pϕ​\(yEmo∣pool⁡\(𝐳P\)\)\.\\displaystyle=\-w\_\{e\}\\log p\_\{\\phi\}\\left\(y\_\{\\mathrm\{Emo\}\}\\mid\\operatorname\{pool\}\(\\mathbf\{z\}\_\{P\}\)\\right\)\.\(13\)Here,yphy\_\{\\mathrm\{ph\}\}denotes the transcript\-derived phone target sequence,yspky\_\{\\mathrm\{spk\}\}andyEmoy\_\{\\mathrm\{Emo\}\}denote the utterance\-level speaker and emotion labels, andyProsy\_\{\\mathrm\{Pros\}\}contains frame\-level log\-F0F\_\{0\}and mean\-centred log\-energy\. The operatorpool⁡\(⋅\)\\operatorname\{pool\}\(\\cdot\)denotes temporal mean pooling, andwew\_\{e\}is the emotion\-class weight\.

These supervision signals alone do not guarantee disentanglement\. A phone adversary therefore acts on𝐳P\\mathbf\{z\}\_\{P\}, while speaker, emotion, and prosody adversaries act on𝐳L\\mathbf\{z\}\_\{L\}\. The gradient reversal layer acts as the identity during the forward pass and multiplies the backward gradient by−γ\-\\gamma\[[7](https://arxiv.org/html/2610.10865#bib.bib3)\]:

GRLγ⁡\(𝐳\)=𝐳,∂GRLγ∂𝐳=−γ​𝐈\.\\operatorname\{GRL\}\_\{\\gamma\}\(\\mathbf\{z\}\)=\\mathbf\{z\},\\qquad\\frac\{\\partial\\operatorname\{GRL\}\_\{\\gamma\}\}\{\\partial\\mathbf\{z\}\}=\-\\gamma\\mathbf\{I\}\.\(14\)
For brevity, let𝐚PR=GRLγPR⁡\(𝐳P\)\\mathbf\{a\}\_\{\\mathrm\{PR\}\}=\\operatorname\{GRL\}\_\{\\gamma\_\{\\mathrm\{PR\}\}\}\(\\mathbf\{z\}\_\{P\}\)and𝐚t=GRLγt⁡\(𝐳L\)\\mathbf\{a\}\_\{t\}=\\operatorname\{GRL\}\_\{\\gamma\_\{t\}\}\(\\mathbf\{z\}\_\{L\}\)fort∈\{SID,Pros,Emo\}t\\in\\\{\\mathrm\{SID,Pros,Emo\}\\\}\. The adversarial losses are:

ℒadv,PR\(P\)\\displaystyle\\mathcal\{L\}^\{\(P\)\}\_\{\\mathrm\{adv,PR\}\}=CTC⁡\(hadv,PR​\(𝐚PR\),yph\),\\displaystyle=\\operatorname\{CTC\}\(h\_\{\\mathrm\{adv,PR\}\}\(\\mathbf\{a\}\_\{\\mathrm\{PR\}\}\),y\_\{\\mathrm\{ph\}\}\),\(15\)ℒadv,SID\(L\)\\displaystyle\\mathcal\{L\}^\{\(L\)\}\_\{\\mathrm\{adv,SID\}\}=−log⁡pψ​\(yspk∣pool⁡\(𝐚SID\)\),\\displaystyle=\-\\log p\_\{\\psi\}\(y\_\{\\mathrm\{spk\}\}\\mid\\operatorname\{pool\}\(\\mathbf\{a\}\_\{\\mathrm\{SID\}\}\)\),\(16\)ℒadv,Pros\(L\)\\displaystyle\\mathcal\{L\}^\{\(L\)\}\_\{\\mathrm\{adv,Pros\}\}=ProsMSE⁡\(hadv,Pros​\(𝐚Pros\),yPros\),\\displaystyle=\\operatorname\{ProsMSE\}\(h\_\{\\mathrm\{adv,Pros\}\}\(\\mathbf\{a\}\_\{\\mathrm\{Pros\}\}\),y\_\{\\mathrm\{Pros\}\}\),\(17\)ℒadv,Emo\(L\)\\displaystyle\\mathcal\{L\}^\{\(L\)\}\_\{\\mathrm\{adv,Emo\}\}=−welogpψ\(yEmo∣\\displaystyle=\-w\_\{e\}\\log p\_\{\\psi\}\(y\_\{\\mathrm\{Emo\}\}\\midOPENhadv,Emo​\(pool⁡\(𝐚Emo\)\)\)\.\\displaystyle\\quad h\_\{\\mathrm\{adv,Emo\}\}\(\\operatorname\{pool\}\(\\mathbf\{a\}\_\{\\mathrm\{Emo\}\}\)\)\)\.\(18\)The total training objective is

ℒtrain=λrecon​ℒrecon\+λAuxK​ℒAuxK\+ℒtask\+ℒadv,\\mathcal\{L\}\_\{\\mathrm\{train\}\}=\\lambda\_\{\\mathrm\{recon\}\}\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\+\\lambda\_\{\\mathrm\{AuxK\}\}\\mathcal\{L\}\_\{\\mathrm\{AuxK\}\}\+\\mathcal\{L\}\_\{\\mathrm\{task\}\}\+\\mathcal\{L\}\_\{\\mathrm\{adv\}\},\(19\)withℒrouting\\mathcal\{L\}\_\{\\mathrm\{routing\}\}added during learned routing\. Since the masks are complementary,𝐳t=𝐳L\+𝐳P\\mathbf\{z\}\_\{t\}=\\mathbf\{z\}\_\{L\}\+\\mathbf\{z\}\_\{P\}, so the complete sparse code continues to reconstruct the encoder representation\.

## 3Experimental Setup

### 3\.1Data, Encoders, and Configurations

The experiments use LibriSpeechtrain\-clean\-100\[[15](https://arxiv.org/html/2610.10865#bib.bib18)\]and MSP\-Podcast\[[4](https://arxiv.org/html/2610.10865#bib.bib19)\]\. Phone labels are obtained from transcripts using the CMU pronunciation lexicon with a grapheme\-to\-phoneme fallback, producing a 74\-symbol phone vocabulary\. On LibriSpeech, PR usesdev\-cleanfor validation andtest\-cleanfor evaluation\. Closed\-set SID uses utterance\-disjoint splits withintrain\-clean\-100, with the same 251 speakers across splits\. MSP\-Podcast evaluates linguistic information through PR, and paralinguistic information through SID, emotion, and prosody\.

We evaluate frozen SPEAR XLarge\[[23](https://arxiv.org/html/2610.10865#bib.bib11)\]and WavLM Large\[[5](https://arxiv.org/html/2610.10865#bib.bib2)\]\. For both encoders, parameter\-free LayerNorm is applied per layer before uniform averaging\. The upstream encoders remain frozen throughout SAE training, and all runs use the same architecture, routing, optimisation settings\. All SAEs use dictionary widthK=5120K=5120, TopKk=256k=256, AuxK budget 64, and AuxK weight1/321/32\. Fixed routing uses contiguousKL/KP=4096/1024K\_\{L\}/K\_\{P\}=4096/1024assigned units and active budgetskL/kP=240/16k\_\{L\}/k\_\{P\}=240/16, selected in preliminary experiments from the validation trade\-off between intended\-factor retention and cross\-factor suppression\.

Under quota\-freeze routing, assignments are learned until step 4,000 and the mean active split over 20 sampled training batches sets the post\-freeze budgets\. The SPEAR reconstruction\-only and fixed\-routing models are trained for 12,000 steps\. The corresponding WavLM configurations use 12,000 steps, except quota\-freeze with 16,000 steps\. We additionally evaluate Eta\-WavLM, which fits an affine speaker component from ECAPA\-TDNN embeddings and subtracts it from the WavLM representation\[[20](https://arxiv.org/html/2610.10865#bib.bib23)\]\.

### 3\.2Evaluation Protocol

All probes are trained post hoc on frozen representations, with parameters entirely separate from the SAE, routing masks, and training\-time adversaries\. Relying on the training\-time adversary as evidence that information has been removed is unreliable\[[6](https://arxiv.org/html/2610.10865#bib.bib15)\]\. PR is evaluated using phone error rate \(PER\), SID using accuracy\. Emotion is evaluated using unweighted average recall \(UAR\), while prosody is evaluated usingF0F\_\{0\}and energy correlations\. Reconstruction fidelity is measured by the mean\-squared error between𝐱t\\mathbf\{x\}\_\{t\}and𝐱^t\\hat\{\\mathbf\{x\}\}\_\{t\}\. For cross\-corpus evaluation, the frozen LibriSpeech\-trained models are evaluated on a 51,017\-utterance MSP\-Podcast subset with 700 speakers and 41,067/4,975/4,975 train/validation/test utterances; closed\-set SID chance is0\.143%0\.143\\%\[[4](https://arxiv.org/html/2610.10865#bib.bib19)\]\. The SAE encoder and decoder, routing masks, and quotas remain frozen; only MSP\-specific phone and speaker probes are trained\.

For controlled intervention, we use a fixed manifest of 250 held\-out recipient–donor utterance pairs with different speakers and at most 10% duration difference, reused across both routing models\. The donor route is linearly interpolated over normalised utterance time to match the recipient’s number of frames\. APP\-route swap combines the recipient’s𝐳L\\mathbf\{z\}\_\{L\}with the resampled donor𝐳P\\mathbf\{z\}\_\{P\}, while anLL\-route swap combines the resampled donor𝐳L\\mathbf\{z\}\_\{L\}with the recipient’s𝐳P\\mathbf\{z\}\_\{P\}\. The recombined code is passed through the frozen SAE decoder, producing a reconstructed SPEAR feature𝐡^\\hat\{\\mathbf\{h\}\}with the recipient’s frame count\. Independent phone and speaker evaluators, fitted only on disjoint unswapped reconstructions, operate on𝐡^\\hat\{\\mathbf\{h\}\}\. We report the percentage of samples retaining the recipient phone after aPP\-swap, changing from the recipient phone after anLL\-swap, or specifically matching the donor phone after anLL\-swap\. Speaker\-match scores report the percentage of samples predicted as the donor speaker after aPP\-swap or retained as the recipient speaker after anLL\-swap\.

Table 1:Independent probes across encoders and corpora\. Reconstruction MSE \(Recon\.\) is reported once per SAE model\.

## 4Results

### 4\.1Linguistic and Paralinguistic Separation

Table[1](https://arxiv.org/html/2610.10865#S3.T1)shows factor\-specific retention and suppression across encoders, corpora, and tasks\. All SAE models are trained either on LibriSPeech or MSP\-Podcast\. The reconstruction\-only SAEs preserve linguistic and paralinguistic information, but sparsity alone does not produce the selected factorisation\. Under both SPEAR and WavLM, phone information remains stronger in𝐳L\\mathbf\{z\}\_\{L\}, while speaker identity remains stronger in𝐳P\\mathbf\{z\}\_\{P\}\. The MSP\-Podcast results show the same route organisation across the broader task set\. Phone information remains stronger in𝐳L\\mathbf\{z\}\_\{L\}, while speaker identity, emotion, and prosody are more recoverable from𝐳P\\mathbf\{z\}\_\{P\}\. Across encoders, fixed routing provides stronger paralinguistic suppression in𝐳L\\mathbf\{z\}\_\{L\}, whereas quota\-freeze more strongly suppresses linguistic information in𝐳P\\mathbf\{z\}\_\{P\}and provides a more balanced reconstruction and route\-selectivity trade\-off\. In the separate Eta\-WavLM evaluation, the affine residual largely preserves phone performance but provides weaker factor\-specific suppression than the routed representations\.

Beyond the models trained directly on each corpus, we additionally test whether the LibriSpeech\-trained SPEAR routing can transfer to MSP\-Podcast without fine\-tuning\. Fixed𝐳L/𝐳P\\mathbf\{z\}\_\{L\}/\\mathbf\{z\}\_\{P\}yields 43\.92/99\.72% PER and 12\.82/82\.71% SID accuracy, while quota\-freeze gives 44\.56/90\.38% PER and 3\.00/82\.29% SID accuracy, both retaining the opposing route pattern under corpus shift\.

### 4\.2Controlled Route Interventions

Table[2](https://arxiv.org/html/2610.10865#S4.T2)shows a similar double dissociation under both routing configurations\. Swapping𝐳P\\mathbf\{z\}\_\{P\}preserves recipient phone information while transferring donor speaker identity, whereas swapping𝐳L\\mathbf\{z\}\_\{L\}transfers donor phone information while preserving recipient speaker identity\. The swap therefore provides strong evidence that the routes are functionally distinct rather than merely statistically distinct\.

Table 2:Controlled route interventions on the fixed 250\-pair SPEAR evaluation set \(%\)\.
### 4\.3Ablations

Table[3](https://arxiv.org/html/2610.10865#S4.T3)shows that task supervision retains the desired information\. Without adversaries,𝐳L\\mathbf\{z\}\_\{L\}retains strong phone performance and𝐳P\\mathbf\{z\}\_\{P\}retains near\-ceiling speaker accuracy\. However, cross\-factor information also remains recoverable\. The speaker adversary reduces speaker recoverability from𝐳L\\mathbf\{z\}\_\{L\}, while the phone adversary reduces phonetic recoverability from𝐳P\\mathbf\{z\}\_\{P\}\. The two provide complementary suppression\.

Table 3:Ablations of the adversarial objectives on SPEAR\. Lower𝐳L\\mathbf\{z\}\_\{L\}SID and higher𝐳P\\mathbf\{z\}\_\{P\}PER indicate stronger cross\-factor suppression\.

## 5Conclusion

This paper investigated the disentanglement of pretrained speech representations using SAE\. With suitable inductive biases, an SAE can produce strong route\-selective separation\. Route\-specific task supervision, cross\-factor adversaries, and a stable routing boundary separate the SAE latent units into linguistic and paralinguistic routes with factor\-specific retention and suppression\. Independent probes show that each route retains the intended information, while cross\-factor recoverability is substantially reduced\. This pattern is observed with both SPEAR XLarge and WavLM Large and is retained under cross\-corpus evaluation on MSP\-Podcast\. Route swapping further shows that the learned routes have distinct functional roles\.

## References

- \[1\]G\. Aparin, T\. Sadekova, A\. Rukhovich,et al\.\(2026\)AudioSAE: towards understanding of audio\-processing models with sparse AutoEncoders\.InProc\. EACL,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[2\]A\. Baevski, H\. Zhou, A\. Mohamed, and M\. Auli\(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InProc\. NeurIPS,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p1.1)\.
- \[3\]T\. Bricken, A\. Templeton, J\. Batson,et al\.\(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[4\]C\. Busso, R\. Lotfian, K\. Sridhar, A\. N\. Salman, W\. Lin, L\. Goncalves, S\. Parthasarathy, A\. R\. Naini, S\. Leem, L\. Martinez\-Lucas, H\. Chou, and P\. Mote\(2026\)The MSP\-Podcast corpus\.IEEE Transactions on Affective Computing\.External Links:[Document](https://dx.doi.org/10.1109/TAFFC.2026.3678489)Cited by:[§3\.1](https://arxiv.org/html/2610.10865#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.10865#S3.SS2.p1.1)\.
- \[5\]S\. Chen, C\. Wang, Z\. Chen,et al\.\(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p1.1),[§1](https://arxiv.org/html/2610.10865#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.10865#S3.SS1.p2.1)\.
- \[6\]Y\. Elazar and Y\. Goldberg\(2018\)Adversarial removal of demographic attributes from text data\.InProc\. EMNLP,Cited by:[§3\.2](https://arxiv.org/html/2610.10865#S3.SS2.p1.1)\.
- \[7\]Y\. Ganin, E\. Ustinova, H\. Ajakan, P\. Germain, H\. Larochelle, F\. Laviolette, M\. Marchand, and V\. Lempitsky\(2016\)Domain\-adversarial training of neural networks\.Journal of Machine Learning Research17\(59\),pp\. 1–35\.External Links:[Link](https://jmlr.org/papers/v17/15-239.html)Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.10865#S2.SS3.p3.1)\.
- \[8\]L\. Gao, T\. Dupré la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu\(2025\)Scaling and evaluating sparse autoencoders\.InProc\. ICLR,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.10865#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.10865#S2.SS1.p1.3)\.
- \[9\]W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed\(2021\)HuBERT: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM Transactions on Audio, Speech, and Language Processing29,pp\. 3451–3460\.Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p1.1)\.
- \[10\]E\. Jang, S\. Gu, and B\. Poole\(2017\)Categorical reparameterization with gumbel\-softmax\.InProc\. ICLR,Cited by:[§2\.2](https://arxiv.org/html/2610.10865#S2.SS2.p2.1)\.
- \[11\]P\. Leask, B\. Bussmann, M\. T\. Pearce,et al\.\(2025\)Sparse autoencoders do not find canonical units of analysis\.InProc\. ICML,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.10865#S2.SS2.p1.1)\.
- \[12\]H\. Lu, X\. Wu, Z\. Wu, and H\. Meng\(2023\)SpeechTripleNet: end\-to\-end disentangled speech representation learning for content, timbre and prosody\.InProc\. ACMMM,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[13\]A\. Makhzani and B\. J\. Frey\(2014\)kk\-sparse autoencoders\.InProc\. ICLR,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.10865#S2.SS1.p1.1)\.
- \[14\]T\. Mariotte, M\. Lebourdais, A\. Almudévar, M\. Tahon, A\. Ortega, and N\. Dugué\(2026\)Sparse autoencoders make audio foundation models more explainable\.InProc\. ICASSP,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[15\]V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur\(2015\)LibriSpeech: an ASR corpus based on public domain audio books\.InProc ICASSP,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.10865#S3.SS1.p1.1)\.
- \[16\]I\. Parra\(2025\)Interpretable sparse features for probing self\-supervised speech models\.InProc\. IJCNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-srw.1),[Link](https://aclanthology.org/2025.ijcnlp-srw.1/)Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[17\]K\. Peng, R\. Movva, J\. Kleinberg, E\. Pierson, and N\. Garg\(2026\)Position: use sparse autoencoders to discover unknowns\.InProc\. ICML,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.10865#S2.SS2.p1.1)\.
- \[18\]K\. Qian, Y\. Zhang, S\. Chang, D\. Cox, and M\. Hasegawa\-Johnson\(2020\)Unsupervised speech decomposition via triple information bottleneck\.InProc\. ICML,External Links:[Link](https://proceedings.mlr.press/v119/qian20a.html)Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[19\]K\. Qian, Y\. Zhang, H\. Gao, J\. Ni, C\. Lai, D\. Cox, M\. Hasegawa\-Johnson, and S\. Chang\(2022\)ContentVec: an improved self\-supervised speech representation by disentangling speakers\.InProc\. ICML,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[20\]G\. Ruggiero, M\. Testa, J\. Van den Walle, and L\. Di Caro\(2025\)Eta\-WavLM: efficient speaker identity removal in self\-supervised speech representations using a simple linear equation\.InFindings of ACL,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.10865#S3.SS1.p3.1)\.
- \[21\]J\. Wang, J\. Li, X\. Zhao, Z\. Wu, S\. Kang, and H\. Meng\(2021\)Adversarially learning disentangled speech representations for robust multi\-factor voice conversion\.InProc\. Interspeech,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p2.1)\.
- \[22\]S\. Yang, H\. Chang, Z\. Huang,et al\.\(2024\)A large\-scale evaluation of speech foundation models\.IEEE/ACM Transactions on Audio, Speech, and Language Processing32,pp\. 2884–2899\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2024.3389631)Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p1.1)\.
- \[23\]X\. Yang, Y\. Yang, Z\. Jin, Z\. Cui, W\. Wu, B\. Li, C\. Zhang, and P\. Woodland\(2026\)SPEAR: a unified SSL framework for learning speech and audio representations\.InProc\. ICML,Cited by:[§1](https://arxiv.org/html/2610.10865#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.10865#S3.SS1.p2.1)\.