MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions

# A Dual Framework for Concept Erasure and Counterfactual Interventions
Source: [https://arxiv.org/html/2609.11253](https://arxiv.org/html/2609.11253)
Antoine Saillenfesta\.saillenfest@groupeonepoint\.com

###### Abstract

Erasing concept\-specific information from representations has been proven useful for mitigating bias or interpreting model decisions\. The joint objective is to transform the original representations such that the target concept becomes unpredictable, while maximally preserving concept\-unrelated information\. In this work, we revisit the optimal bounds of concept erasure to derive a novel class of erasure functions that naturally induce a deterministic, dual counterfactual mapping\. Bridging the gap between theoretical optimality and practical representation learning, we design an implementation that imposes a translational bias on counterfactual trajectories—a constraint that aligns with how many concepts geometrically manifest in modern language models\. Our framework enables seamless navigation between concept erasure and counterfactual generation\. We empirically demonstrate its efficacy in improving downstream algorithmic fairness and generating counterfactual texts\.111Code and data:[https://github\.com/toinesayan/MUtE](https://github.com/toinesayan/MUtE)\.

## 1Introduction

Text representations inherently entangle a diverse array of latent concepts, ranging from sensitive demographic attributes \(e\.g\. gender or race\) requiring algorithmic mitigation to abstract semantic properties \(e\.g\. writing style or aspect\) manipulated for interpretability\. Because targeted interventions in the discrete textual space are difficult to automate and frequently yield disfluent artifacts, representation engineering predominantly isolates and manipulates these concepts directly within the continuous embedding space\.

The specific task of concept erasure aims to obliterate information pertaining to a target concept from a set of representations via an erasure function\([Ravfogel et al\., 2020](https://arxiv.org/html/2609.11253#bib.bib25);[Ravfogel et al\., 2022a](https://arxiv.org/html/2609.11253#bib.bib27);[Chowdhury et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib11)\)\. From an information\-theoretic perspective, optimal erasure necessitates navigating a fundamental trade\-off: maximizing privacy \(ensuring the post\-erasure representations contain minimal information regarding the target concept\) while maximizing utility \(maximizing the retained information from the original representations\)\([Chowdhury et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib11)\)\. Because concept erasure is fundamentally task\-agnostic, the resulting representations are broadly applicable across downstream NLP tasks, demonstrating efficacy in improving algorithmic fairness\([Ravfogel et al\., 2020](https://arxiv.org/html/2609.11253#bib.bib25);[Chowdhury and Chaturvedi, 2022](https://arxiv.org/html/2609.11253#bib.bib9);[Lemberger and Saillenfest, 2024](https://arxiv.org/html/2609.11253#bib.bib20)\), enhancing interpretability\([Lemberger and Saillenfest, 2024](https://arxiv.org/html/2609.11253#bib.bib20)\), and mitigating language model toxicity\([Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\)\.

![Refer to caption](https://arxiv.org/html/2609.11253v1/assets/intro_image_final.png)Figure 1:TheMUtE∗\\text\{MUtE\}^\{\*\}continuous erasure framework\. A piecewise erasure functionffutilizes class\-conditional diffeomorphismsfif\_\{i\}to bijectively map source distributionsPiP\_\{i\}to a shared invariant space, maximizing utility retention\. This structure natively induces a dual counterfactual mapping \(f2−1∘f1f\_\{2\}^\{\-1\}\\circ f\_\{1\}\), anchoring a samplexxand its counterfactualx2←1x\_\{2\\leftarrow 1\}to the identical post\-erasure representationf⁡\(x\)f\(x\)\. Constraining these counterfactual trajectories with ana prioristructural prior—such as the linear translational bias prevalent in NLP—ties the conditional mappings, effectively resolving the functional non\-identifiability of optimal erasure\.Driven by these applications, concept erasure has seen rapid development\. By relaxing strict privacy constraints, a significant body of work has addressed linear guardedness\([Ravfogel et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib28)\), which consists in preventing concept recovery by a linear adversary\. Methods such as INLP\([Ravfogel et al\., 2020](https://arxiv.org/html/2609.11253#bib.bib25)\), RLACE\([Ravfogel et al\., 2022a](https://arxiv.org/html/2609.11253#bib.bib27)\), and the state\-of\-the\-art LEACE\([Belrose et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib6)\)achieve linear erasure via projections onto a low\-dimensional concept nullspace\. However, general \(non\-linear\) concept erasure remains an open challenge\. Current approaches include adversarial training\([Ganin et al\., 2016](https://arxiv.org/html/2609.11253#bib.bib15);[Feder et al\., 2021](https://arxiv.org/html/2609.11253#bib.bib14);[Ravfogel et al\., 2022b](https://arxiv.org/html/2609.11253#bib.bib26)\), information\-theoretic loss optimization \(FaRM\([Chowdhury and Chaturvedi, 2022](https://arxiv.org/html/2609.11253#bib.bib9)\), KRaM\([Chowdhury et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib10)\)\), Bayesian optimization \(PEF\([Chowdhury et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib11)\)\), filtering of ranked directions out of embeddings \(TaCo[Jourdan et al\. \(2024\)](https://arxiv.org/html/2609.11253#bib.bib18)\) and density matching via projections \(L¯\\overline\{\\text\{L\}\}EOPARD\([Saillenfest and Lemberger, 2025](https://arxiv.org/html/2609.11253#bib.bib29)\)\)\. Those approaches, however, frequently sacrifice downstream utility to satisfy strict privacy bounds\. Furthermore, these methods suffer from severe computational bottlenecks, relying on extensive neural network training, expensive density estimations, or Bayesian optimization over finite supports, which scales poorly in high\-dimensional regimes\.

Concurrently, recent literature has established profound connections between concept erasure and the generation of counterfactual representations, i\.e\. approximations of the embedding space under simulated causal interventions on the surface text\. Assuming a structural causal model and Gaussian priors,[Lemberger and Saillenfest \(2024\)](https://arxiv.org/html/2609.11253#bib.bib20)have demonstrated that concept erasure can act as an intermediary mapping for counterfactual generation\. Inspired by linear erasure techniques, recent works derive linear steering interventions under minimal displacement constraints\([Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\), which have been decoded into counterfactual texts using continuous\-to\-discrete inversion techniques\([Avitan et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib5);[Morris et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib21)\)\.

In this work, we revisit discrete concept erasure within continuous representation spaces at optimality\. We formalize the structure of optimal erasure functions and demonstrate that they inherently induce a dual, deterministic counterfactual mapping\. Bridging theory and practice, we propose a practical implementation that imposes a rigid translational bias on counterfactual trajectories\. This approach practically exploits the phenomenon that modern text encoders frequently isolate concepts as linear directions\.

To summarize, our primary contributions are:

- •We introduceMUtE∗\(MaximumUtility\-preservingErasure\), a class of optimal erasure functions for continuous representations that intrinsically defines a dual counterfactual mapping \(Section[2](https://arxiv.org/html/2609.11253#S2)\)\.
- •We derive a practical, computationally efficient implementation that leverages a translational bias to perform erasure and counterfactual representation generation \(Section[3](https://arxiv.org/html/2609.11253#S3)\)\.
- •We empirically validate our framework across synthetic data and NLP benchmarks, demonstrating its efficacy for bias mitigation and counterfactual text generation \(Section[4](https://arxiv.org/html/2609.11253#S4)\)\.

## 2Optimal Erasure and Counterfactuals

This section formalizes the theoretical foundations of Maximum Utility\-preserving Erasure \(MUtE∗\\text\{MUtE\}^\{\*\}\)\. After defining the problem setting \(Section[2\.1](https://arxiv.org/html/2609.11253#S2.SS1)\), we review the necessary conditions for perfect erasure \(Section[2\.2](https://arxiv.org/html/2609.11253#S2.SS2)\) and maximal utility retention \(Section[2\.3](https://arxiv.org/html/2609.11253#S2.SS3)\)\. Assuming an oracle predictor, we demonstrate that this optimal erasure naturally induces a dual counterfactual mapping \(Section[2\.4](https://arxiv.org/html/2609.11253#S2.SS4)\), providing a principled geometric mechanism to resolve the inherent non\-identifiability of erasure functions\.

### 2\.1Problem statement

Let𝐏𝒮\\mathbf\{P\}\_\{\\mathcal\{S\}\}be a probability distribution over an arbitrary input space𝒮\\mathcal\{S\}\. Letenc:𝒮→ℝD\\mathrm\{enc\}:\\mathcal\{S\}\\to\\mathbb\{R\}^\{D\}be an encoding function mapping inputs to a continuous representation space, and let𝒵=\{1,…,K\}\\mathcal\{Z\}=\\\{1,\\dots,K\\\}\(K\>1K\>1\) denote a discrete set of concept classes assigned by a ground\-truth labeling functionc∗:𝒮→𝒵c^\{\*\}:\\mathcal\{S\}\\to\\mathcal\{Z\}\.

We define the concept variableZZand the representation variableX∼𝐏XX\\sim\\mathbf\{P\}\_\{X\}as random variables over𝒮\\mathcal\{S\}:

Z​\(s\)=c∗​\(s\)Z\(s\)=c^\{\*\}\(s\)\(1\)X⁡\(s\)=enc⁡\(s\)X\(s\)=\\mathrm\{enc\}\(s\)\(2\)
Letp⁡\(x\)p\(x\)denote the probability density function \(PDF\) of𝐏X\\mathbf\{P\}\_\{X\}\. For each concept classi∈𝒵i\\in\\mathcal\{Z\}, we introduce the class\-conditional representation variableXi∼𝐏iX\_\{i\}\\sim\\mathbf\{P\}\_\{i\}, where the corresponding conditional density is defined aspi​\(x\):=p⁡\(x∣Z=i\)p\_\{i\}\(x\):=p\(x\\mid Z=i\)\.

Concept erasure seeks anerasure functionffthat removes concept\-related information aboutZZ\(Eq\.[1](https://arxiv.org/html/2609.11253#S2.E1)\) from the representationsXX\(Eq\.[2](https://arxiv.org/html/2609.11253#S2.E2)\)\. We define the post\-erasure representation asQ=f⁡\(X\)Q=f\(X\), distributed according to the pushforward measure𝐐=f\#​𝐏X\\mathbf\{Q\}=f\_\{\\\#\}\\mathbf\{P\}\_\{X\}, with corresponding class\-conditional distributions𝐐i\\mathbf\{Q\}\_\{i\}\.

We make the following assumptions:

- •The erasure function preserves the latent dimensionality, i\.e\.,f:ℝD→ℝDf:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}\.
- •The conditional distributions𝐏i\\mathbf\{P\}\_\{i\}are continuous overℝD\\mathbb\{R\}^\{D\}and exhibit full support \(i\.e\., there is no zero\-density regions\)\.222In our primary NLP setting \(𝒮=Σ∗\\mathcal\{S\}=\\Sigma^\{\*\}, whereΣ\\Sigmais an alphabet and𝐏𝒮\\mathbf\{P\}\_\{\\mathcal\{S\}\}is a language model\), discrete textual inputs strictly map to point masses inℝD\\mathbb\{R\}^\{D\}\. To satisfy continuity and full\-support constraints, we assume these discrete representations are continuously smoothed \(e\.g\., via kernel density estimation\)\.

### 2\.2Discrete Concept Erasure\.

Concept erasure fundamentally seeks a transformationffthat enforces strict statistical independence betweenf⁡\(X\)f\(X\)andZZ\. This translates into a rigorous distributional constraint: the class\-conditional distributions of the post\-erasure representations must be perfectly aligned\([Saillenfest and Lemberger, 2025](https://arxiv.org/html/2609.11253#bib.bib29)\)\. Formally:

∀i∈𝒵,𝐐i=𝐐\\forall i\\in\\mathcal\{Z\},\\quad\\mathbf\{Q\}\_\{i\}=\\mathbf\{Q\}\(3\)
From an information\-theoretic perspective, Eq\.[3](https://arxiv.org/html/2609.11253#S2.E3)corresponds to achieving the privacy bound characterized by vanishing mutual information:I⁡\(f⁡\(X\),Z\)=0I\(f\(X\);Z\)=0\([Chowdhury et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib11)\)\.

Crucially, however, this pure erasure objective is not self\-sufficient\. An infinite set of mappings satisfies this independence criterion, including degenerate, constant functions \(e\.g\.,f⁡\(x\)=k∈ℝDf\(x\)=k\\in\\mathbb\{R\}^\{D\}\) that catastrophically collapse the representation space\. Consequently, any viable concept erasure framework must formulate a joint objective: satisfying the erasure criterion while maximizing the retention of task\-agnostic information \(utility\) inherent to the original representations\.

### 2\.3Maximum Utility Preservation\.

From an information\-theoretic perspective, preserving maximum utility requires the conditional entropy to vanish:H⁡\(X\|f⁡\(X\),Z\)=0H\(X\|f\(X\),Z\)=0\([Chowdhury et al\., 2025](https://arxiv.org/html/2609.11253#bib.bib11)\)\. Conditioning on a concept classi∈𝒵i\\in\\mathcal\{Z\}, the equalityH⁡\(Xi\|f⁡\(Xi\)\)=0H\(X\_\{i\}\|f\(X\_\{i\}\)\)=0implies thatXiX\_\{i\}can be perfectly recovered from its post\-erasure representation\. Since the forward mapping is deterministic by definition \(H⁡\(f⁡\(Xi\)\|Xi\)=0H\(f\(X\_\{i\}\)\|X\_\{i\}\)=0\), this mutual determinism ensures thatffmaps each conditional distribution𝐏i\\mathbf\{P\}\_\{i\}bijectively to𝐐i\\mathbf\{Q\}\_\{i\}\.

Structurally, integrating the perfect erasure constraint \(Eq\.[3](https://arxiv.org/html/2609.11253#S2.E3)\),MUtE∗\(Maximum Utility\-preserving Erasure\) functions satisfies:

∀s∈𝒮,f⁡\(X⁡\(s\)\)=fc∗​\(s\)​\(X⁡\(s\)\)\\forall s\\in\\mathcal\{S\},\\quad f\(X\(s\)\)=f\_\{c^\{\*\}\(s\)\}\(X\(s\)\)\(4\)where eachfi:ℝD→ℝDf\_\{i\}:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\}is a diffeomorphism transforming𝐏i\\mathbf\{P\}\_\{i\}into𝐐\\mathbf\{Q\}\.

This continuous\-space formulation mirrors the optimal erasure framework introduced by[Chowdhury et al\. \(2025\)](https://arxiv.org/html/2609.11253#bib.bib11)\. Their approach achieves optimal privacy\-utility bounds via class\-conditional permutations, assuming discrete supports of equal cardinality that are identical up to permutation relative to the target distribution\. Our framework generalizes these optimality principles to continuous probability measures by replacing combinatorial permutations with diffeomorphisms\.

The sample\-dependent assignmentc∗​\(s\)c^\{\*\}\(s\)in Eq\.[4](https://arxiv.org/html/2609.11253#S2.E4)precludesfffrom operating solely on the observable representationXX\. To circumvent this limitation,[Chowdhury et al\. \(2025\)](https://arxiv.org/html/2609.11253#bib.bib11)assume disjoint conditional supports, guaranteeing perfect predictability of the concept class from the representation alone\. In practice, high\-dimensional continuous representations inevitably exhibit overlapping supports, yielding an irreducible Bayes error that precludes strict determinism\. However, because modern latent spaces demonstrate high structural separability, routinely allowing probes to achieve near\-perfect empirical classification, we adopt a tractable theoretical surrogate\. We assume the existence of an optimal oracle predictorη∗\\eta^\{\*\}that recovers the concept class with negligible error, idealizing it as exact for our formulation:

∀s∈𝒮,η∗​\(X⁡\(s\)\)=Z⁡\(s\)=c∗​\(s\)\\forall s\\in\\mathcal\{S\},\\quad\\eta^\{\*\}\(X\(s\)\)=Z\(s\)=c^\{\*\}\(s\)\(5\)
Under this oracle assumption,MUtE∗\\text\{MUtE\}^\{\*\}functions \(Eq\.[4](https://arxiv.org/html/2609.11253#S2.E4)\) can be expressed as a class of representation\-dependent erasure mappings:

f:ℝD→ℝD,x↦fη∗​\(x\)​\(x\)f:\\mathbb\{R\}^\{D\}\\to\\mathbb\{R\}^\{D\},\\quad x\\mapsto f\_\{\\eta^\{\*\}\(x\)\}\(x\)\(6\)
Because concept erasure enforces alignment strictly at the distributional level \(Eq\.[3](https://arxiv.org/html/2609.11253#S2.E3)\), the hypothesis space of validMUtE∗\\text\{MUtE\}^\{\*\}mappings is infinite\. This permits arbitrary topological distortions of the representation space\. To resolve this fundamental non\-identifiability, we next exploit the dual counterfactual mapping intrinsically induced by these optimal erasure functions\.

### 2\.4Dual Counterfactual Mapping

Let the counterfactual mapping of a representation\-dependent MUtE∗functionff\(Eq\.[6](https://arxiv.org/html/2609.11253#S2.E6)\) be:

f←:ℝD×𝒵→ℝD,\(x,j\)↦fj−1​\(f⁡\(x\)\)f\_\{\\leftarrow\}:\\mathbb\{R\}^\{D\}\\times\\mathcal\{Z\}\\to\\mathbb\{R\}^\{D\},\\quad\(x,j\)\\mapsto f\_\{j\}^\{\-1\}\(f\(x\)\)\(7\)
This transport is termedcounterfactualbecause, for any distinct concept classesi,j∈𝒵i,j\\in\\mathcal\{Z\}, it mapsXi∼𝐏iX\_\{i\}\\sim\\mathbf\{P\}\_\{i\}toXj←i:=f←​\(Xi,j\)=fj−1​\(fi​\(Xi\)\)X\_\{j\\leftarrow i\}:=f\_\{\\leftarrow\}\(X\_\{i\},j\)=f\_\{j\}^\{\-1\}\(f\_\{i\}\(X\_\{i\}\)\)that is strictly distributed according to𝐏j\\mathbf\{P\}\_\{j\}\.

By definition,f⁡\(Xj←i\)=f⁡\(Xi\)f\(X\_\{j\\leftarrow i\}\)=f\(X\_\{i\}\)\. This establishes a fundamental structural correspondence between a MUtE∗function and the counterfactuals it generates: a sample and all its counterfactual counterparts map to the exact same invariant coordinate in the post\-erasure latent space\. Consequently, each erased representation serves as a geometric anchor connecting\|𝒵\|\|\\mathcal\{Z\}\|counterfactual representations\.

Crucially, this collision property provides a principled mechanism to resolve the inherent non\-identifiability of optimal erasure functions\. If the geometry of the target counterfactualsXj←iX\_\{j\\leftarrow i\}can be approximateda priori, it directly constrains the hypothesis space offfby tying the diffeomorphismsfif\_\{i\}andfjf\_\{j\}\. By enforcingf⁡\(Xj←i\)=f⁡\(Xi\)f\(X\_\{j\\leftarrow i\}\)=f\(X\_\{i\}\)over these expected counterfactual trajectories, we can isolate geometrically grounded solutions from the infinite space of otherwise valid MUtE∗functions\.

In NLP, the geometric relationship between representations and their counterfactuals has been extensively studied\. The Linear Bias Hypothesis posits that deep NLP models naturally encode concepts as linear directions within their high\-dimensional latent spaces\([Bolukbasi et al\., 2016](https://arxiv.org/html/2609.11253#bib.bib8);[Vargas and Cotterell, 2020](https://arxiv.org/html/2609.11253#bib.bib33)\)\. Thus, recent steering techniques operationalize counterfactual generation via linear interventions\([Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\)\. The simplest such intervention approximates a counterfactual shift by translating the representation along the vector difference between class centroids\([Subramani et al\., 2022](https://arxiv.org/html/2609.11253#bib.bib32);[Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\)\.

Synthesizing the theoretical formulation above, Figure[1](https://arxiv.org/html/2609.11253#S1.F1)illustrates theMUtE∗\\text\{MUtE\}^\{\*\}framework, demonstrating how its invertible mappings enable bidirectional geometric navigation between concept erasure and counterfactual generation\.

## 3Operationalizing MUtE∗via Iterative Density Matching

This section bridges the theoretical formulation ofMUtE∗\\text\{MUtE\}^\{\*\}with a computationally tractable implementation\. We first recall an iterative Gaussianization procedure \(Section[3\.1](https://arxiv.org/html/2609.11253#S3.SS1)\), adapting it to construct a fully representation\-dependent erasure mapping \(Section[3\.2](https://arxiv.org/html/2609.11253#S3.SS2)\)\. Geometrically, this mapping imposes an inherent translational bias on the counterfactual trajectories\. We then relax the assumption of an oracle predictorη∗\\eta^\{\*\}to formally characterize the impact of noisy routing\. This theoretical analysis directly motivates algorithmic regularizations to promote effective concept erasure within deeply entangled latent spaces \(Section[3\.3](https://arxiv.org/html/2609.11253#S3.SS3)\)\.

### 3\.1Iterative Gaussianization

Rotation\-Based Iterative Gaussianization \(RBIG\)\([Laparra et al\., 2011](https://arxiv.org/html/2609.11253#bib.bib19)\)is a highly efficient, iterative Gaussianization technique that transforms any continuous random distribution𝐏\\mathbf\{P\}into an isotropic Gaussian𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)\. For a set of observationsxx, the process alternates between applying an orthogonal rotation matrixR\(t\)∈O⁡\(D\)R^\{\(t\)\}\\in\\mathrm\{O\}\(D\)and a dimension\-wise marginal Gaussianizationψ\(t\)\\psi^\{\(t\)\}:

x\(t\+1\)=\(ψ\(t\)∘R\(t\)\)​\(x\(t\)\)withx\(0\)=x\\begin\{array\}\[\]\{l\}x^\{\(t\+1\)\}=\\left\(\\psi^\{\(t\)\}\\circ R^\{\(t\)\}\\right\)\(x^\{\(t\)\}\)\\\\ \\text\{with\}\\quad x^\{\(0\)\}=x\\end\{array\}\(8\)
where:

ψ\(t\)​\(x\(t\)\)=\(Φ−1​\(∫−∞xd\(t\)pd\(t\)​\(u\)​𝑑u\)\)d=1,…,D\\psi^\{\(t\)\}\(x^\{\(t\)\}\)=\\left\(\\Phi^\{\-1\}\\left\(\\int\_\{\-\\infty\}^\{x^\{\(t\)\}\_\{d\}\}p^\{\(t\)\}\_\{d\}\(u\)du\\right\)\\right\)\_\{d=1,\\dots,D\}\(9\)
The marginal Gaussianizationψ\(t\)\\psi^\{\(t\)\}operates independently on each dimensiondd, mapping the data to𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)via a marginal uniformization, based on the cumulative density function \(CDF\) of the marginal probability density function \(PDF\)pdp\_\{d\}, followed by the inverse CDF of the standard normal𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\),Φ−1\\Phi^\{\-1\}\.

Provided the sequence of orthogonal rotations\{R\(t\)\}\\\{R^\{\(t\)\}\\\}induces sufficient cross\-dimensional mixing \(e\.g\., via orthogonal ICA\([Hyvarinen et al\., 2019](https://arxiv.org/html/2609.11253#bib.bib16)\), PCA\([Jolliffe and Cadima, 2016](https://arxiv.org/html/2609.11253#bib.bib17)\), or random orthogonal matrices\), RBIG guarantees monotonic convergence to𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)\. This monotonic convergence is formally characterized by a strict reduction in negentropy, defined here as the Kullback\-Leibler divergence to the standard isotropic Gaussian, at steptt\. The negentropy reductionΔ​J\(t\)\\Delta J^\{\(t\)\}at stepttis thus strictly positive:

ΔJ\(t\):=DKL\(𝐏\(t\)∥𝒩\(0,I\)\)−DKL\(𝐏\(t\+1\)∥𝒩\(0,I\)\)\>0\\Delta J^\{\(t\)\}:=D\_\{\\mathrm\{KL\}\}\(\\mathbf\{P\}^\{\(t\)\}\\parallel\\mathcal\{N\}\(0,I\)\)\-D\_\{\\mathrm\{KL\}\}\(\\mathbf\{P\}^\{\(t\+1\)\}\\parallel\\mathcal\{N\}\(0,I\)\)\>0\(10\)
The overall Gaussianization process is bijective\. Rotations are invertible and\(R\(t\)\)−1=\(R\(t\)\)⊤\(R^\{\(t\)\}\)^\{\-1\}=\(R^\{\(t\)\}\)^\{\\top\}\.ψ\(t\)\\psi^\{\(t\)\}is invertible when the support of each marginal PDF is connected \(i\.e\. there are no zero\-probability regions\) making the marginal CDF strictly monotonic and hence invertible\.

### 3\.2Selective Iterative Density Matching

We adapt the process defined in Eq\.[8](https://arxiv.org/html/2609.11253#S3.E8)into a conditional mapping, selective based on the predicted classη∗​\(x\)\\eta^\{\*\}\(x\)of the samplexx:

x\(t\+1\)=\(ψη∗​\(x\)\(t\)∘R\(t\)\)​\(x\(t\)\)withx\(0\)=x\\begin\{array\}\[\]\{l\}x^\{\(t\+1\)\}=\\left\(\\psi\_\{\\eta^\{\*\}\(x\)\}^\{\(t\)\}\\circ R^\{\(t\)\}\\right\)\(x^\{\(t\)\}\)\\\\ \\text\{with\}\\quad x^\{\(0\)\}=x\\end\{array\}\(11\)
whereR\(t\)R^\{\(t\)\}is an orthogonal rotation, andψi\(t\)\\psi\_\{i\}^\{\(t\)\}denotes the class\-conditional, dimension\-wise marginal Gaussianization at steptt\. For a given concept classii,ψi\(t\)\\psi\_\{i\}^\{\(t\)\}is defined using the marginal class\-conditional PDFpi,d\(t\)p\_\{i,d\}^\{\(t\)\}for the dimensiondd:

ψi\(t\)​\(x\)=\(Φ−1​\(∫−∞xdpi,d\(t\)​\(u\)​𝑑u\)\)d=1,…,D\\psi\_\{i\}^\{\(t\)\}\(x\)=\\left\(\\Phi^\{\-1\}\\left\(\\int\_\{\-\\infty\}^\{x\_\{d\}\}p\_\{i,d\}^\{\(t\)\}\(u\)du\\right\)\\right\)\_\{d=1,\\dots,D\}\(12\)
At each iteration,R\(t\)R^\{\(t\)\}is applied uniformly across the entire representation space, acting as an isometry that preserves the macroscopic geometric structure\. Conversely, the class\-conditional marginal transformationsψi\(t\)\\psi\_\{i\}^\{\(t\)\}independently warp the conditional densities to align their marginals with a standard normal distribution\.

Under the assumption thatη∗\\eta^\{\*\}acts as a perfect oracle \(Eq\.[5](https://arxiv.org/html/2609.11253#S2.E5)\) and conditioned on any concept classii, this process mirrors the standard RBIG, which guarantees that𝐏i\\mathbf\{P\}\_\{i\}is bijectively mapped to an isotropic Gaussian𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)ast→∞t\\to\\infty\. Consequently, this iterative procedure asymptotically drives all conditional distributions𝐏i\\mathbf\{P\}\_\{i\}to a shared target measure, theoretically guaranteeing complete concept erasure after a sufficient number of iterationsTT\.333Concept erasure requires only that the class\-conditional densities match \(Eq\.[3](https://arxiv.org/html/2609.11253#S2.E3)\), not full convergence to𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)\. In practice,TTdenotes the number of steps to satisfy Eq\.[3](https://arxiv.org/html/2609.11253#S2.E3)\.

Truncating the iterative process at stepTTyields an empirical conditional\-bijectiveMUtEfunction:

f:ℝd→ℝd,x↦fη∗​\(x\)​\(x\)with​fi=ψi\(T−1\)∘R\(T−1\)∘⋯∘ψi\(0\)∘R\(0\)\\begin\{array\}\[\]\{l\}f:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\},\\quad x\\mapsto f\_\{\\eta^\{\*\}\(x\)\}\(x\)\\\\ \\text\{with \}f\_\{i\}=\\psi\_\{i\}^\{\(T\-1\)\}\\circ R^\{\(T\-1\)\}\\circ\\dots\\circ\\psi\_\{i\}^\{\(0\)\}\\circ R^\{\(0\)\}\\end\{array\}\(13\)
Crucially, the initial marginal Gaussianizations\(ψi\(0\)\)i=1,…,\|𝒵\|\(\\psi\_\{i\}^\{\(0\)\}\)\_\{i=1,\\dots,\|\\mathcal\{Z\}\|\}independently map the median of each conditional distribution to the origin\. Because deep continuous representations typically exhibit symmetric, Gaussian\-like marginals where the mean and median closely align, this zero\-order matching acts as a rigid location shift\. Consequently, this dominant first step imparts a strong geometric inductive bias, steering counterfactual representations predominantly along the inter\-centroid vector𝔼⁡\[Xj\]−𝔼⁡\[Xi\]\\mathbb\{E\}\[X\_\{j\}\]\-\\mathbb\{E\}\[X\_\{i\}\]\. The subsequent sequence of global rotations and marginal transformations non\-linearly refines this trajectory to ensure exact higher\-order distributional alignment without destroying the foundational translational bias\.

### 3\.3Implementation and Mitigation of Noisy Routing

Due to overlapping conditional supports, the empirical predictorη∗\\eta^\{\*\}inherently exhibits classification error\. Crucially,MUtEdecouples density estimation from sample routing: marginal transformationsψi\(t\)\\psi\_\{i\}^\{\(t\)\}are fitted using ground\-truth labels, while the forward mapping of a samplexxrelies strictly on its predicted classη∗​\(x\)\\eta^\{\*\}\(x\)\.

This irreducible Bayes error theoretically precludes strict convergence to an exact isotropic Gaussian\. We formalize this limitation by expressing the stepwise negentropy reduction for classiiunder noisy routing \(see Appendix[A](https://arxiv.org/html/2609.11253#A1)for the full derivation\):

Δ​Ji\(t\)=Δ​Ji∗\(t\)−Ei\(t\)\\Delta J\_\{i\}^\{\(t\)\}=\\Delta J^\{\*\(t\)\}\_\{i\}\-E^\{\(t\)\}\_\{i\}\(14\)
whereΔ​Ji∗\(t\)≥0\\Delta J^\{\*\(t\)\}\_\{i\}\\geq 0denotes the ideal negentropy reduction at stepttunder perfect routing, andEi\(t\)\>0E^\{\(t\)\}\_\{i\}\>0represents a strictly positive entropic penalty induced by misrouting\. Consequently, the convergence of each class\-conditional distribution to𝒩⁡\(0,I\)\\mathcal\{N\}\(0,I\)is fundamentally bottlenecked by this routing error\.

Nevertheless, this analytical decomposition directly motivates algorithmic regularizations to optimize the stepwise negentropy reduction—specifically, by maximizing the ideal marginal gainΔ​Ji∗\(t\)\\Delta J^\{\*\(t\)\}\_\{i\}and bounding the routing penaltyEi\(t\)E^\{\(t\)\}\_\{i\}\. Although exact asymptotic convergence to the standard normal is mathematically unattainable, we expect that enforcing these bounds iteratively drives the conditional manifolds into a sufficiently tight, shared neighborhood, to effectively neutralizes class separability\.

To maximize the marginal Gaussianization gainΔ​Ji∗\(t\)\\Delta J^\{\*\(t\)\}\_\{i\}, the rotation sequence\{R\(t\)\}\\\{R^\{\(t\)\}\\\}must systematically induce sufficient cross\-dimensional mixing\. While techniques such as orthogonal ICA drive rapid convergence, PCA offers a superior trade\-off between step\-wise negentropy reduction and computational complexity\([Laparra et al\., 2011](https://arxiv.org/html/2609.11253#bib.bib19)\)\. To satisfy this mixing requirement concurrently across all concept classesi∈𝒵i\\in\\mathcal\{Z\}, we adopt an alternating rotation schedule:R\(t\)=RiPCAR^\{\(t\)\}=R\_\{i\}^\{\\text\{PCA\}\}, wherei≡t\(mod\|𝒵\|\)i\\equiv t\\pmod\{\|\\mathcal\{Z\}\|\}andRiPCAR\_\{i\}^\{\\text\{PCA\}\}diagonalizes the covariance of𝐏i\(t\)\\mathbf\{P\}\_\{i\}^\{\(t\)\}\. This cyclic scheme periodically aligns the principal axes of each conditional manifold, ensuring their unique cross\-correlations are successively exposed for marginal Gaussianization\. Although a rotationRiPCAR\_\{i\}^\{\\text\{PCA\}\}optimized for classiidoes not explicitly target the cross\-correlations of a distinct classj≠ij\\neq i, the RBIG framework guarantees that any valid orthogonal rotation still yields a non\-negative negentropy reduction \(Δ​Jj∗\(t\)≥0\\Delta J^\{\*\(t\)\}\_\{j\}\\geq 0\)\. Moreover, if the class\-conditional manifolds are approximately equivalent up to a translation, their covariance structures inherently align\. In this regime, rotating byRiPCAR\_\{i\}^\{\\text\{PCA\}\}is thus expected to simultaneously drive a high negentropy reduction for all classes at every iterative step\.

The routing penaltyEi\(t\)E^\{\(t\)\}\_\{i\}quantifies the entropic cost of spatial tearing, driven by the cross\-entropy mismatch when a sample is evaluated via the marginal density estimator of a mispredicted class\. To maintain focus on the practical mitigation strategy, we defer the full analytical expansion and bounding ofEi\(t\)E^\{\(t\)\}\_\{i\}to Appendix[A\.2](https://arxiv.org/html/2609.11253#A1.SS2)\. Crucially, this penalty diverges to infinity if the true density is strictly positive where the mispredicted estimator assigns zero probability\. To bound this log\-density ratio, we estimate the continuous conditional PDFspi,d\(t\)p\_\{i,d\}^\{\(t\)\}via marginal histograms uniformly discretized intoBBbins, enforcing a strict minimum density thresholdpi,d\(t\)​\(xd\)≥α\>0p\_\{i,d\}^\{\(t\)\}\(x\_\{d\}\)\\geq\\alpha\>0\. This threshold constrainsEi\(t\)E^\{\(t\)\}\_\{i\}with an upper bound\. By defaulting toα=10−10\\alpha=10^\{\-10\}andB=1000B=1000, the artificial probability mass injected across the domain \(B​α=10−7B\\alpha=10^\{\-7\}\) remains negligible\.

While these regularizations cannot theoretically guarantee perfect distributional alignment afterTTiterations, our subsequent evaluations on real\-world datasets demonstrate that this constrained implementation yields highly robust concept erasure in practice\.

## 4Experiments

We validate our approach on synthetic and NLP datasets \(Section[4\.1](https://arxiv.org/html/2609.11253#S4.SS1)\) for concept erasure and utility preservation \(Section[4\.2](https://arxiv.org/html/2609.11253#S4.SS2)\), bias mitigation \(Section[4\.3](https://arxiv.org/html/2609.11253#S4.SS3)\) and counterfactual generation of texts \(Section[4\.4](https://arxiv.org/html/2609.11253#S4.SS4)\)\.

### 4\.1Datasets, Baselines and Training details

Table 1:Key dataset statistics\.yydenotes the availability of a downstream classification task\.#### Baselines\.

We compare our approach with several baselines for erasure: FaRM\([Chowdhury and Chaturvedi, 2022](https://arxiv.org/html/2609.11253#bib.bib9)\), KRaM\([Chowdhury et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib10)\), TaCo\([Jourdan et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib18)\), andL¯\\overline\{\\mathrm\{L\}\}EOPARD\([Saillenfest and Lemberger, 2025](https://arxiv.org/html/2609.11253#bib.bib29)\), and the linear erasure method LEACE\([Belrose et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib6)\)for completeness\. Training details are deferred to Appendix[C](https://arxiv.org/html/2609.11253#A3)\.

#### NLP benchmarks\.

We evaluate our approach for concept erasure across a suite of text embeddings: gender fromGloVe\([Pennington et al\., 2014](https://arxiv.org/html/2609.11253#bib.bib24)\), gender from BERT representations\([Devlin et al\., 2019](https://arxiv.org/html/2609.11253#bib.bib13)\)of short biographies inBias in Bios\([De\-Arteaga et al\., 2019](https://arxiv.org/html/2609.11253#bib.bib12)\), race from DeepMoji representations ofDIALtweets\([Blodgett et al\., 2016](https://arxiv.org/html/2609.11253#bib.bib7)\), and religion from GPT\-4 embeddings\([Achiam et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib2)\)of online comments inJigsaw\([, 2019](https://arxiv.org/html/2609.11253#bib.bib1)\)\. Table[1](https://arxiv.org/html/2609.11253#S4.T1)summarizes key statistics, other details are deferred to Appendix[B](https://arxiv.org/html/2609.11253#A2)\.

#### Evaluation\.

Probes for concept prediction and downstream classification were implemented as MLPs \(scikit\-learn’sMLPClassifier\([Pedregosa et al\., 2011](https://arxiv.org/html/2609.11253#bib.bib23)\)\)\. Mean Squared Error \(MSE\) estimations were conducted usingMLPRegressor\. Accuracies, MSEs, and fairness scores reported are averages across55independent evaluations\.

![Refer to caption](https://arxiv.org/html/2609.11253v1/assets/synthetic_data.png)Figure 2:Concept erasure viaMUtEon synthetic distributions exhibiting near\-perfect separability \(top row\) versus moderate overlap \(bottom row\)\. Background hues indicate decision regions for the routing predictor \(cols\. 1, 3, 4\) and the optimal adversarial probe \(col\. 2\)\. Point colors denote ground\-truth concept labels\.![Refer to caption](https://arxiv.org/html/2609.11253v1/assets/utility_privacy_tradeoff_large.png)Figure 3:Probing accuracy\-Utility tradeoff\. Horizontal red \(resp\. dotted\) lines are chance\-level baseline accuracies \(resp\. accuracies on the original latent space\)\.

### 4\.2Erasure and utility preservation

#### Synthetic data\.

We evaluate MUtE on syntheticℝ2\\mathbb\{R\}^\{2\}data, modeling a binary concept using two different Gaussian mixtures of2,0002,000samples each\. We control concept separability by translating the mixtures along a fixed vector, defining two regimes with initial probe accuracies of94%94\\%and81%81\\%, respectively \(Figure[2](https://arxiv.org/html/2609.11253#S4.F2), left\)\. In both settings, we train MUtE for2020iterations\. Post\-erasure, the conditional distributions are aligned, and probe accuracy drops to chance \(Figure[2](https://arxiv.org/html/2609.11253#S4.F2), second column\)\. In both regimes, the mapping produces artifacts corresponding to misclassified samples; their counterfactual displacement is roughly inverse to the trajectory expected under ground\-truth routing \(Figure[2](https://arxiv.org/html/2609.11253#S4.F2), third column\)\. The counterfactual mapping exhibits a strong translational bias, confirming our geometric priors \(Figure[2](https://arxiv.org/html/2609.11253#S4.F2), right\)\.

#### Real\-world datasets

We extend our evaluation of MUtE to real\-world NLP benchmarks\. Results in Figure[3](https://arxiv.org/html/2609.11253#S4.F3)indicates that MUtE consistently reduces probe accuracy for sensitive attributes to the majority\-class baseline \(e\.g\.,50%50\\%for balanced binary concepts\) consistently outperforming baseline methods\. Following[Chowdhury et al\. \(2025\)](https://arxiv.org/html/2609.11253#bib.bib11), we use the MSE of reconstructingXXfromf⁡\(X\)f\(X\)as a surrogate for utility preservation, noting that MUtE maintains low reconstruction error\. Furthermore, MUtE demonstrates competitve or higher performance \(accuracyaya\_\{y\}\) on downstream tasks \(Table[2](https://arxiv.org/html/2609.11253#S4.T2)\)\.

ForGloVe, we quantify the preservation of intrinsic semantic and local topological properties\. On the WordSim\-353 benchmark, we measure the Spearman rank correlation between embedding cosine similarities and human annotations\([Agirre et al\., 2009](https://arxiv.org/html/2609.11253#bib.bib3)\)\. The original embeddings yield a correlation of0\.700\.70\. MUtE successfully preserves this semantic alignment \(0\.710\.71\), whereas baseline methods degrade it \(FaRM:0\.53\\text\{FaRM\}:0\.53,KRaM:0\.33\\text\{KRaM\}:0\.33\)\. We also evaluate local topological fidelity by computing the retention rate of the top1%1\\%nearest neighbors post\-erasure\. MUtE retains19%19\\%of the original neighborhood structure, outperforming bothFaRM\(13%13\\%\) andKRaM\(10%10\\%\)\.

Table 2:Fairness evaluation for downstream classifiers\. Best results \(excluding original representations\) in bold\.

### 4\.3Fair classification

A primary downstream application of concept erasure is mitigating algorithmic bias to improve fairness\. Following established methodologies\([Ravfogel et al\., 2020](https://arxiv.org/html/2609.11253#bib.bib25);[De\-Arteaga et al\., 2019](https://arxiv.org/html/2609.11253#bib.bib12);[Saillenfest and Lemberger, 2025](https://arxiv.org/html/2609.11253#bib.bib29)\), we evaluate fairness using two metrics suited for binary concepts: the Root Mean Square of the True Positive Rate Gap \(TPRRMS\\mathrm\{TPR\}^\{\\mathrm\{RMS\}\}\) and Demographic Parity \(DP\) \(those metrics are formally defined in Appendix[D](https://arxiv.org/html/2609.11253#A4)\)\. Results in Table[2](https://arxiv.org/html/2609.11253#S4.T2)shows that downstream classifiers trained on MUtE\-transformed representations exhibit substantial fairness improvements compared to those trained on the original latent space\. MUtE is competitive with established baselines, establishing a distinct operating point on the accuracy\-fairness Pareto frontier that strongly favors the preservation of representation utility forBias in Bios\.

The dual counterfactual mapping induced by MUtE enable its application as a data augmentation technique to train fair classifiers directly within the original representation space\. Specifically, we train downstream classifiers onBias in BiosandDIALusing 50k original samples paired with their 50k generated counterfactuals, yielding a balanced 100k\-sample training corpus\. We evaluate two distinct supervisory regimes: \(1\) Oracle\-Supervised \(MUtE∗←\{\}^\{\*\}\_\{\\leftarrow\}\): Concept and task labels are jointly available during training, allowing us to generate exact counterfactual representations using the theoretical MUtE∗function, and \(2\) Disjoint\-Supervised \(MUtE←\): Concept and task labels reside in mutually exclusive training sets, MUtE is optimized strictly on the concept\-annotated partition and subsequently deployed to augment the task\-annotated partition\. We benchmark these approaches against two latent steering interventions for counterfactual generation: class\-conditional centroid translation \(Mean Diff\.\)\([Subramani et al\., 2022](https://arxiv.org/html/2609.11253#bib.bib32);[Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\)and class\-conditional linear optimal transport \(Linear OT\)\([Singh et al\., 2024](https://arxiv.org/html/2609.11253#bib.bib31)\)\. As shown in Table[3](https://arxiv.org/html/2609.11253#S4.T3), counterfactual augmentation via MUtE∗←\{\}^\{\*\}\_\{\\leftarrow\}and MUtE←drives significant fairness gains in the original representation space, consistently rivaling or outperforming linear steering techniques\.

Table 3:Fairness evaluation after data augmentation\. Best results \(excluding original representations\) in bold\.

### 4\.4Counterfactual text generation

We evaluate the quality of counterfactual representations generated byMUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}against linear steering baselines in the context of discrete text generation\. Using the continuous\-to\-discrete text inversion framework proposed by[Morris et al\. \(2023\)](https://arxiv.org/html/2609.11253#bib.bib21), we project the counterfactual embeddings back into the natural language space\. Specifically, we train a 4\-iterationMUtE∗\\text\{MUtE\}^\{\*\}model on GTR embeddings\([Ni et al\., 2022](https://arxiv.org/html/2609.11253#bib.bib22)\)derived from theBias in Biosdataset\. For test sequences under3232tokens \(N=628N=628, bounded by the maximum sequence length observed during the inversion model’s training\), we generate counterfactual embeddings usingMUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}, Mean Difference, and Linear OT, which are subsequently decoded into text\. To isolate the impact of the optimization objective from the functional capacity of the mapping, we also train a class\-conditional linear surrogate ofMUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}via linear regressions to approximate the counterfactual mappings \(male←\\leftarrowfemale and female←\\leftarrowmale\)\. We benchmark the decoded texts against ground\-truth textual counterfactuals—constructed via rule\-based substitutions\([De\-Arteaga et al\., 2019](https://arxiv.org/html/2609.11253#bib.bib12)\)—using BLEU and ROUGE \(1/2/LF1F\_\{1\}\-scores\) to measure lexical overlap, and BERTScore \(F1F\_\{1\}\) to quantify dense semantic preservation\. Finally, to evaluate the efficacy of the latent intervention, we define thegender substitution rate\(rgr\_\{\\text\{g\}\}\) as the proportion of generated gender indicators \(he, him, his, mr, m, himselfandshe, her, hers, ms, mrs, herself\) that successfully match the target counterfactual class\.

Qualitative examples of the inverted mapping,MUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}\(Table[4](https://arxiv.org/html/2609.11253#S4.T4)\), demonstrate effective gender substitution with high semantic fidelity\. Quantitatively \(Table[5](https://arxiv.org/html/2609.11253#S4.T5)\), although texts decoded fromMUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}exhibit lowernn\-gram overlap \(BLEU/ROUGE\) with the source text compared to baselines, they sustain high BERTScores and achieve a significantly superior gender substitution rate\. This discrepancy exposes a fundamental failure mode of standard linear steering baselines: their high lexical overlap is largely an artifact of under\-intervention\. Operating essentially as near\-identity functions, these methods artificially inflatenn\-gram metrics by passively reconstructing the original text\. In contrast, by enforcing high\-order alignment of the conditional distributions,MUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}induces deep structural alterations to execute a robust semantic intervention, while preserving the core semantic utility\. Notably, the linear surrogate derived fromMUtE←∗\\text\{MUtE\}^\{\*\}\_\{\\leftarrow\}achieves comparable efficacy, thereby introducing a novel, computationally efficient linear steering mechanism for targeted concept intervention\.

Table 4:Counterfactual texts decoded after intervention on gender via MUtE∗←\{\}^\{\*\}\_\{\\leftarrow\}and Linear OT steering on theBias in Biosdataset\. Original gender markers arebolded\. Post\-intervention markers that successfully align with the target gender are highlighted in green and bolded, whereas contradictory or unchanged markers are highlighted in red and italic\.
ModelGenderTextoriginalmaleHereceivedhisBA in Mathematics Education from Mercyhurst College in Eric, Pennsylvania, andhisPhD in Mathematics from the University of South Carolina\.Linear OTfemaleHereceivedherBA in Mathematics Education from Mercyhurst College in Eric, Pennsylvania, andherPhD in Mathematics from the University of South Carolina, Tri Carolina\.MUtE∗←\{\}^\{\*\}\_\{\\leftarrow\}femaleShereceivedherBA in Mathematics Education from Mercyhurst College in South Carolina, andherPhD in the Department of Mathematics and Education at Eric University in Pennsylvania\.originalfemaleSheparticipated in projects concerning data analysis\.Herresearch interests include applications of game theory to queueing networks, and to inventory management\.Linear OTmaleSheparticipated in projects related to data analysis\.Hisresearch interests include applications of game theory to inventory management, data management, and queueing networks \.MUtE∗←\{\}^\{\*\}\_\{\\leftarrow\}maleHeparticipated on projects related to data analysis\.Hisresearch interests include applications of game theory to inventory management, and stochastic queueing game networks\.Table 5:Counterfactual text generation evaluation\.True CFR\(CounterFactual Representation\) corresponds to texts decoded from the embeddings of true counterfactual text\.

## 5Discussion and Future Directions

This work builds upon recent frameworks unifying concept erasure and counterfactual representation generation, demonstrating they are fundamentally dual interventions\. By formalizing the navigation between erased and counterfactual continuous spaces via diffeomorphisms, our approach establishes a versatile paradigm that readily extends beyond NLP\.

Our approach bypasses gradient\-based optimization, yielding an efficient, deterministic training phase in high\-dimensional continuous spaces\. This training efficiency, however, trades off against inference latency: the forward mapping cost scales linearly with the number of iterationsTTrequired for convergence\. To circumvent the𝒪⁡\(T\)\\mathcal\{O\}\(T\)inference bottleneck, the exact multi\-step mappingffcan be distilled into an amortized surrogatefθf\_\{\\theta\}by minimizing the regression lossℒ=𝔼⁡\[‖f⁡\(X\)−fθ​\(X\)‖22\]\{\\mathcal\{L\}=\\mathbb\{E\}\\left\[\\\|f\(X\)\-f\_\{\\theta\}\(X\)\\\|\_\{2\}^\{2\}\\right\]\}\. This offline distillation seamlessly accommodates diverse deployment constraints\. To strictly preserve the bijectivity required for dual counterfactual generation,fθf\_\{\\theta\}can be parameterized using class\-specific Invertible Neural Networks \(INNs\) or memory\-efficient Conditional INNs \(cINNs\)\([Ardizzone et al\., 2020](https://arxiv.org/html/2609.11253#bib.bib4)\)\. When solely forward erasure is required, relaxing the bijectivity constraint via standard feed\-forward networks \(e\.g\., MLPs\) maximizes inference throughput\.

As a preliminary proof\-of\-concept, we empirically demonstrate in Appendix[E](https://arxiv.org/html/2609.11253#A5)that a MLP surrogate successfully preserves both the predictive utility and fairness guarantees of the exactMUtErepresentations on downstream tasks\. This confirms that a lightweight feed\-forward network can adequately approximate the underlying transformations, offering an efficient alternative for practical deployment\. We leave extensive evaluations of these distillation strategies to future work\.

Beyond resolving computational limitations, subsequent research must expand the framework’s causal expressivity\. While our current empirical estimator enforces a rigid translational bias, real\-world causal factors of variation frequently interact via complex, non\-linear mechanisms exhibiting heterogeneous geometric signatures\([Schölkopf et al\., 2021](https://arxiv.org/html/2609.11253#bib.bib30)\)\. Formulating and integrating expressive geometric priors to capture these diverse, non\-linear causal interventions remains an open challenge that necessitates novel theoretical approaches and dedicated future research\.

## 6Conclusion

This work revisits the problem of discrete concept erasure at optimality, formally defined as achieving perfect privacy while maximizing downstream utility preservation\. We derive a class of theoretically optimal erasure functions that naturally induce a dual, deterministic counterfactual mapping within the continuous representation space\. To bridge theory and practice, we propose a computationally tractable implementation that aligns with this theoretical framework\. We demonstrate its empirical efficacy across real\-world Natural Language Processing tasks, successfully exploiting the geometric phenomenon that many latent concepts in modern language models manifest as rigid location shifts\. Notably, our method proves highly effective for both algorithmic bias mitigation and the generation of counterfactual texts\. Future work will explore extending this framework to other data modalities such as image representations in computer vision, and integrating complex geometric priors to model diverse causal interventions\.

## Limitations

Our framework is strictly formulated for the erasure of a single, discrete concept\. It does not naturally extend to continuous sensitive attributes—a regime where methods such as FaRM\([Chowdhury and Chaturvedi, 2022](https://arxiv.org/html/2609.11253#bib.bib9)\)and KRaM\([Chowdhury et al\., 2023](https://arxiv.org/html/2609.11253#bib.bib10)\)are currently better suited—nor does it directly accommodate the simultaneous joint erasure of multiple intersecting concepts\.

While enforcing a translational geometric prior is highly effective for many latent concepts in modern text encoders, real\-world causal factors of variation frequently interact via highly non\-linear mechanisms\. Consequently, applying MUtE to arbitrarily complex, entangled representations without verifying the underlying topological assumptions risks suboptimal or unpredictable interventions\.

## Ethical Considerations

In this work, we intervene on sensitive demographic attributes—such as gender, race, and religion—by formulating them as discrete categorical variables drawn from a restricted label set\. In real\-world applications, operationalizing these attributes requires rigorous consensus, a process frequently hindered by heterogeneous cultural, ethical, and legal contexts\. Furthermore, the forced erasure of concept\-related representations inherently induces a loss of information\. This utility degradation can severely impair downstream predictive performance, posing significant risks when deploying these models in high\-stakes domains \(e\.g\., healthcare, criminal justice, or resource allocation\)\. Consequently, algorithmic fairness cannot be the sole evaluation criterion in practice\. Finally, targeted interventions on specific data dimensions carry unintended systemic risks: erasing a single sensitive attribute may inadvertently incentivize the model to exploit unprotected proxy variables, potentially exacerbating representation biases against intersecting demographic groups\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1)\.
- Agirreet al\.\(2009\)E\. Agirre, E\. Alfonseca, K\. Hall, J\. Kravalova, M\. Paşca, and A\. SoroaA study on similarity and relatedness using distributional and WordNet\-based approaches\.InProceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics,M\. Ostendorf, M\. Collins, S\. Narayanan, D\. W\. Oard, and L\. Vanderwende \(Eds\.\),Boulder, Colorado,pp\. 19–27\.Cited by:[§4\.2](https://arxiv.org/html/2609.11253#S4.SS2.SSS0.Px2.p2.1)\.
- Ardizzoneet al\.\(2020\)L\. Ardizzone, J\. Kruse, C\. Lüth, N\. Bracher, C\. Rother, and U\. KötheConditional invertible neural networks for diverse image\-to\-image translation\.InDAGM German Conference on Pattern Recognition,pp\. 373–387\.Cited by:[§5](https://arxiv.org/html/2609.11253#S5.p2.1)\.
- Avitanet al\.\(2025\)M\. Avitan, R\. Cotterell, Y\. Goldberg, and S\. RavfogelA practical method for generating string counterfactuals\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 3267–3286\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p4.1)\.
- Belroseet al\.\(2023\)N\. Belrose, D\. Schneider\-Joseph, S\. Ravfogel, R\. Cotterell, E\. Raff, and S\. BidermanLEACE: perfect linear concept erasure in closed form\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px1.p1.1)\.
- Blodgettet al\.\(2016\)S\. L\. Blodgett, L\. Green, and B\. O’ConnorDemographic dialectal variation in social media: a case study of african\-american english\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1119–1130\.Cited by:[Appendix B](https://arxiv.org/html/2609.11253#A2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1)\.
- Bolukbasiet al\.\(2016\)T\. Bolukbasi, K\. Chang, J\. Y\. Zou, V\. Saligrama, and A\. T\. KalaiMan is to computer programmer as woman is to homemaker? debiasing word embeddings\.Advances in neural information processing systems29\.Cited by:[§2\.4](https://arxiv.org/html/2609.11253#S2.SS4.p5.1)\.
- Chowdhury and Chaturvedi \(2022\)S\. B\. R\. Chowdhury and S\. ChaturvediLearning fair representations via rate\-distortion maximization\.Transactions of the Association for Computational Linguistics10,pp\. 1159–1174\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00512)Cited by:[§C\.3](https://arxiv.org/html/2609.11253#A3.SS3.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2609.11253#Sx1.p1.1)\.
- Chowdhuryet al\.\(2025\)S\. B\. R\. Chowdhury, K\. A\. Dubey, A\. Beirami, R\. Kidambi, N\. Monath, A\. Ahmed, and S\. ChaturvediFundamental limits of perfect concept erasure\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Y\. Li, S\. Mandt, S\. Agrawal, and E\. Khan \(Eds\.\),Proceedings of Machine Learning Research, Vol\.258,pp\. 901–909\.External Links:[Link](https://proceedings.mlr.press/v258/chowdhury25a.html)Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.11253#S2.SS2.p3.1),[§2\.3](https://arxiv.org/html/2609.11253#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2609.11253#S2.SS3.p4.1),[§2\.3](https://arxiv.org/html/2609.11253#S2.SS3.p5.1),[§4\.2](https://arxiv.org/html/2609.11253#S4.SS2.SSS0.Px2.p1.1)\.
- Chowdhuryet al\.\(2023\)S\. B\. R\. Chowdhury, N\. Monath, K\. A\. Dubey, A\. Ahmed, and S\. ChaturvediRobust concept erasure via kernelized rate\-distortion maximization\.Advances in Neural Information Processing Systems36\.Cited by:[Appendix B](https://arxiv.org/html/2609.11253#A2.SS0.SSS0.Px3.p1.1),[§C\.3](https://arxiv.org/html/2609.11253#A3.SS3.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2609.11253#Sx1.p1.1)\.
- De\-Arteagaet al\.\(2019\)M\. De\-Arteaga, A\. Romanov, H\. Wallach, J\. Chayes, C\. Borgs, A\. Chouldechova, S\. Geyik, K\. Kenthapadi, and A\. T\. KalaiBias in bios: a case study of semantic representation bias in a high\-stakes setting\.InProceedings of the Conference on Fairness, Accountability, and Transparency,FAT\* ’19,New York, NY, USA,pp\. 120–128\.External Links:ISBN 9781450361255,[Document](https://dx.doi.org/10.1145/3287560.3287572)Cited by:[Appendix B](https://arxiv.org/html/2609.11253#A2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2609.11253#S4.SS3.p1.1),[§4\.4](https://arxiv.org/html/2609.11253#S4.SS4.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL\-HLT 2019,Vol\.1,Minneapolis, MN, USA,pp\. 4171––4186\.Cited by:[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1)\.
- Federet al\.\(2021\)A\. Feder, N\. Oved, U\. Shalit, and R\. ReichartCausalm: causal model explanation through counterfactual language models\.Computational Linguistics47\(2\),pp\. 333–386\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p3.1)\.
- Ganinet al\.\(2016\)Y\. Ganin, E\. Ustinova, H\. Ajakan, P\. Germain, H\. Larochelle, F\. Laviolette, M\. March, and V\. LempitskyDomain\-adversarial training of neural networks\.Journal of machine learning research17\(59\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p3.1)\.
- Hyvarinenet al\.\(2019\)A\. Hyvarinen, H\. Sasaki, and R\. TurnerNonlinear ica using auxiliary variables and generalized contrastive learning\.InProceedings of the Twenty\-Second International Conference on Artificial Intelligence and Statistics,K\. Chaudhuri and M\. Sugiyama \(Eds\.\),Proceedings of Machine Learning Research, Vol\.89,pp\. 859–868\.External Links:[Link](https://proceedings.mlr.press/v89/hyvarinen19a.html)Cited by:[§3\.1](https://arxiv.org/html/2609.11253#S3.SS1.p6.1)\.
- \[16\]\(2019\)Jigsaw unintended bias in toxicity classification\.External Links:[Link](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/data)Cited by:[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1)\.
- Jolliffe and Cadima \(2016\)I\. T\. Jolliffe and J\. CadimaPrincipal component analysis: a review and recent developments\.Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences374\(2065\),pp\. 20150202\.Cited by:[§3\.1](https://arxiv.org/html/2609.11253#S3.SS1.p6.1)\.
- Jourdanet al\.\(2024\)F\. Jourdan, L\. Béthune, A\. Picard, L\. Risser, and N\. AsherTaCo: targeted concept erasure prevents non\-linear classifiers from detecting protected attributes\.arXiv preprint arXiv:2312\.06499\.Cited by:[§C\.3](https://arxiv.org/html/2609.11253#A3.SS3.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px1.p1.1)\.
- Laparraet al\.\(2011\)V\. Laparra, G\. Camps\-Valls, and J\. MaloIterative gaussianization: from ica to random rotations\.IEEE transactions on neural networks22\(4\),pp\. 537–549\.Cited by:[§A\.1](https://arxiv.org/html/2609.11253#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2609.11253#A1.SS1.p3.1),[§3\.1](https://arxiv.org/html/2609.11253#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2609.11253#S3.SS3.p6.1)\.
- Lemberger and Saillenfest \(2024\)P\. Lemberger and A\. SaillenfestExplaining text classifiers with counterfactual representations\.InECAI 2024 \- 27th European Conference on Artificial Intelligence,U\. Endriss, F\. S\. Melo, K\. Bach, A\. J\. B\. Diz, J\. M\. Alonso\-Moral, S\. Barro, and F\. Heintz \(Eds\.\),Frontiers in Artificial Intelligence and Applications, Vol\.392,pp\. 890–897\.External Links:[Document](https://dx.doi.org/10.3233/FAIA240576)Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p4.1)\.
- Morriset al\.\(2023\)J\. Morris, V\. Kuleshov, V\. Shmatikov, and A\. M\. RushText embeddings reveal \(almost\) as much as text\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12448–12460\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p4.1),[§4\.4](https://arxiv.org/html/2609.11253#S4.SS4.p1.1)\.
- Niet al\.\(2022\)J\. Ni, C\. Qu, J\. Lu, Z\. Dai, G\. H\. Abrego, J\. Ma, V\. Zhao, Y\. Luan, K\. Hall, M\. Chang,et al\.Large dual encoders are generalizable retrievers\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 9844–9855\.Cited by:[§4\.4](https://arxiv.org/html/2609.11253#S4.SS4.p1.1)\.
- Pedregosaet al\.\(2011\)F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg,et al\.Scikit\-learn: machine learning in python\.the Journal of machine Learning research12,pp\. 2825–2830\.Cited by:[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px3.p1.1)\.
- Penningtonet al\.\(2014\)J\. Pennington, R\. Socher, and C\. D\. ManningGlove: global vectors for word representation\.InProceedings of the 2014 conference on empirical methods in natural language processing \(EMNLP\),pp\. 1532–1543\.Cited by:[Appendix B](https://arxiv.org/html/2609.11253#A2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px2.p1.1)\.
- Ravfogelet al\.\(2020\)S\. Ravfogel, Y\. Elazar, H\. Gonen, M\. Twiton, and Y\. GoldbergNull it out: guarding protected attributes by iterative nullspace projection\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 7237–7256\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.647)Cited by:[Appendix B](https://arxiv.org/html/2609.11253#A2.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§4\.3](https://arxiv.org/html/2609.11253#S4.SS3.p1.1)\.
- Ravfogelet al\.\(2023\)S\. Ravfogel, Y\. Goldberg, and R\. CotterellLog\-linear guardedness and its implications\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9413–9431\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p3.1)\.
- Ravfogelet al\.\(2022a\)S\. Ravfogel, M\. Twiton, Y\. Goldberg, and R\. D\. CotterellLinear adversarial concept erasure\.InInternational Conference on Machine Learning,pp\. 18400–18421\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1)\.
- Ravfogelet al\.\(2022b\)S\. Ravfogel, F\. Vargas, Y\. Goldberg, and R\. CotterellAdversarial concept erasure in kernel space\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6034–6055\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p3.1)\.
- Saillenfest and Lemberger \(2025\)A\. Saillenfest and P\. LembergerNonlinear concept erasure: a density matching approach\.InECAI 2025 \- 28th European Conference on Artificial Intelligence,I\. Lynce, N\. Murano, M\. Vallati, S\. Villata, F\. Chesani, M\. Milano, A\. Omicini, and M\. Dastani \(Eds\.\),Frontiers in Artificial Intelligence and Applications, Vol\.413,pp\. 2138–2145\.External Links:[Document](https://dx.doi.org/10.3233/FAIA251053)Cited by:[§C\.3](https://arxiv.org/html/2609.11253#A3.SS3.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.11253#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.11253#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.11253#S4.SS1.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2609.11253#S4.SS3.p1.1)\.
- Schölkopfet al\.\(2021\)B\. Schölkopf, F\. Locatello, S\. Bauer, N\. R\. Ke, N\. Kalchbrenner, A\. Goyal, and Y\. BengioToward causal representation learning\.Proceedings of the IEEE109\(5\),pp\. 612–634\.Cited by:[§5](https://arxiv.org/html/2609.11253#S5.p4.1)\.
- Singhet al\.\(2024\)S\. Singh, S\. Ravfogel, J\. Herzig, R\. Aharoni, R\. Cotterell, and P\. KumaraguruRepresentation surgery: theory and practice of affine steering\.InProceedings of the 41st International Conference on Machine Learning,pp\. 45663–45680\.Cited by:[§1](https://arxiv.org/html/2609.11253#S1.p2.1),[§1](https://arxiv.org/html/2609.11253#S1.p4.1),[§2\.4](https://arxiv.org/html/2609.11253#S2.SS4.p5.1),[§4\.3](https://arxiv.org/html/2609.11253#S4.SS3.p2.1)\.
- Subramaniet al\.\(2022\)N\. Subramani, N\. Suresh, and M\. PetersExtracting latent steering vectors from pretrained language models\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 566–581\.External Links:[Link](https://aclanthology.org/2022.findings-acl.48/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.48)Cited by:[§2\.4](https://arxiv.org/html/2609.11253#S2.SS4.p5.1),[§4\.3](https://arxiv.org/html/2609.11253#S4.SS3.p2.1)\.
- Vargas and Cotterell \(2020\)F\. Vargas and R\. CotterellExploring the linear subspace hypothesis in gender bias mitigation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 2902–2913\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.232)Cited by:[§2\.4](https://arxiv.org/html/2609.11253#S2.SS4.p5.1)\.

## Appendix AConditional Negentropy Evolution

In this section, we analyse the evolution of the negentropy for each class of samples\. We first compute the theoretical negentropy reduction under a perfect routing assumption \(Section[A\.1](https://arxiv.org/html/2609.11253#A1.SS1)\)\. Subsequently, we rigorously evaluate the negentropy evolution under the empirical noisy routing assumption \(Section[A\.2](https://arxiv.org/html/2609.11253#A1.SS2)\)\.

Leti∈𝒵i\\in\\mathcal\{Z\}denote a target concept class andttdenote an arbitrary iteration step of MUtE\. LetR\(t\)R^\{\(t\)\}and\{ψk\(t\)\}k∈𝒵\\\{\\psi\_\{k\}^\{\(t\)\}\\\}\_\{k\\in\\mathcal\{Z\}\}be the orthogonal rotation matrix and the class\-conditional marginal Gaussianizations at steptt, respectively\. For notational brevity, we omit the iteration superscript\(t\)\(t\)where unambiguous\.

LetXi\(t\)X\_\{i\}^\{\(t\)\}be a random variable distributed according to the true class\-conditional probability density function \(PDF\)pi\(t\)p\_\{i\}^\{\(t\)\}\. We define the rotated variable asX~i=R​Xi\(t\)\\tilde\{X\}\_\{i\}=RX\_\{i\}^\{\(t\)\}, with corresponding joint PDFp~i\(t\)\\tilde\{p\}\_\{i\}^\{\(t\)\}and marginal PDFsp~i,d\(t\)\\tilde\{p\}\_\{i,d\}^\{\(t\)\}for each dimensiond∈\{1,…,D\}d\\in\\\{1,\\dots,D\\\}\.

We denote the rotated decision regions, partitioned by the potentially noisy empirical predictorη∗\\eta^\{\*\}, asΩ~k=\{x~∈ℝD∣η∗​\(R−1​x~\)=k\}\\tilde\{\\Omega\}\_\{k\}=\\\{\\tilde\{x\}\\in\\mathbb\{R\}^\{D\}\\mid\\eta^\{\*\}\(R^\{\-1\}\\tilde\{x\}\)=k\\\}\. Using the indicator function𝕀Ω~k\\mathbb\{I\}\_\{\\tilde\{\\Omega\}\_\{k\}\}, the piecewise forward mapping of a samplex∼Xi\(t\)x\\sim X\_\{i\}^\{\(t\)\}\(Equation[11](https://arxiv.org/html/2609.11253#S3.E11)\) can be expressed over the rotated space as:

x\(t\+1\)=ψη∗​\(x\)​\(R​x\)=∑k∈𝒵𝕀Ω~k​\(x~\)​ψk​\(x~\)x^\{\(t\+1\)\}=\\psi\_\{\\eta^\{\*\}\(x\)\}\(Rx\)=\\sum\_\{k\\in\\mathcal\{Z\}\}\\mathbb\{I\}\_\{\\tilde\{\\Omega\}\_\{k\}\}\(\\tilde\{x\}\)\\psi\_\{k\}\(\\tilde\{x\}\)
Consequently, the pushforward measure for the entire classiiis given by:

Xi\(t\+1\)=∑k∈𝒵𝕀Ω~k​\(X~i\)​ψk​\(X~i\)X\_\{i\}^\{\(t\+1\)\}=\\sum\_\{k\\in\\mathcal\{Z\}\}\\mathbb\{I\}\_\{\\tilde\{\\Omega\}\_\{k\}\}\(\\tilde\{X\}\_\{i\}\)\\psi\_\{k\}\(\\tilde\{X\}\_\{i\}\)\(15\)
We track the convergence using the negentropyJ\(X\)=DKL\(P∥𝒩\(0,I\)\)J\(X\)=D\_\{\\mathrm\{KL\}\}\(P\\parallel\\mathcal\{N\}\(0,I\)\), which can be decomposed into the differential entropy and a cross\-entropy penalty against the standard normal PDFϕ\\phi\(which is the derivative of the standard normal CDFΦ′​\(x\)=ϕ​\(x\)\\Phi^\{\\prime\}\(x\)=\\phi\(x\)\):

J\(X\)=−h\(X\)−𝔼x∼X\[log∏d=1Dϕ\(xd\)\]J\(X\)=\-h\(X\)\-\\mathbb\{E\}\_\{x\\sim X\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\phi\(x\_\{d\}\)\\right\]\(16\)where the differential entropy is defined as:

h⁡\(X\)=−𝔼x∼X​\[log⁡p⁡\(x\)\]h\(X\)=\-\\mathbb\{E\}\_\{x\\sim X\}\[\\log p\(x\)\]\(17\)
### A\.1Negentropy Reduction Under Perfect Routing

Under the assumption of an optimal oracle predictor \(η∗​\(x\)=i\\eta^\{\*\}\(x\)=i\), all samples belonging to classiiare transformed strictly via the corresponding diffeomorphismψi∘R\\psi\_\{i\}\\circ R\. Following standard Iterative Gaussianization \(RBIG\) principles\([Laparra et al\., 2011](https://arxiv.org/html/2609.11253#bib.bib19)\), each transformation strictly reduces the negentropy of the distribution\. The negentropy reduction under perfect routing is:

Δ​Ji∗=J⁡\(Xi\(t\)\)−J⁡\(\(ψi∘R\)​\(Xi\(t\)\)\)\\Delta J^\{\*\}\_\{i\}=J\(X\_\{i\}^\{\(t\)\}\)\-J\(\(\\psi\_\{i\}\\circ R\)\(X\_\{i\}^\{\(t\)\}\)\)
Because orthogonal rotations are entropy\-preserving isometries,J⁡\(Xi\(t\)\)=J⁡\(X~i\)J\(X\_\{i\}^\{\(t\)\}\)=J\(\\tilde\{X\}\_\{i\}\)\. Thus:

Δ​Ji∗=J⁡\(X~i\)−J⁡\(ψi​\(X~i\)\)\\Delta J^\{\*\}\_\{i\}=J\(\\tilde\{X\}\_\{i\}\)\-J\(\\psi\_\{i\}\(\\tilde\{X\}\_\{i\}\)\)
Recalling property 3\.1 from[Laparra et al\. \(2011\)](https://arxiv.org/html/2609.11253#bib.bib19), this reduction exactly equals the sum of the marginal negentropies of the rotated representations:

Δ​Ji∗=∑d=1D𝔼x~∼X~i​\[log⁡p~i,d\(t\)​\(x~d\)ϕ⁡\(x~d\)\]≥0\\Delta J^\{\*\}\_\{i\}=\\sum\_\{d=1\}^\{D\}\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\frac\{\\tilde\{p\}\_\{i,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\{\\phi\(\\tilde\{x\}\_\{d\}\)\}\\right\]\\geq 0

### A\.2Negentropy Evolution Under Noisy Routing

We now evaluate the evolution of the negentropyΔ​Ji\\Delta J\_\{i\}under a realistic noisy routing regime, where overlaps in conditional distributions yield an irreducible Bayes error\. The exact negentropy reduction can be decomposed as:

Δ​Ji\\displaystyle\\Delta J\_\{i\}=J⁡\(Xi\(t\)\)−J⁡\(Xi\(t\+1\)\)\\displaystyle=J\(X\_\{i\}^\{\(t\)\}\)\-J\(X\_\{i\}^\{\(t\+1\)\}\)\(18\)=ΔJ∗i−\(J⁡\(Xi\(t\+1\)\)−J⁡\(ψi​\(X~i\)\)\)⏟:=Ei\\displaystyle=\\Delta J^\{\*\}\_\{i\}\-\\underbrace\{\\left\(J\(X\_\{i\}^\{\(t\+1\)\}\)\-J\(\\psi\_\{i\}\(\\tilde\{X\}\_\{i\}\)\)\\right\)\}\_\{:=E\_\{i\}\}
To explicitly computeJ⁡\(Xi\(t\+1\)\)J\(X\_\{i\}^\{\(t\+1\)\}\), we first evaluate the differential entropyh⁡\(Xi\(t\+1\)\)h\(X\_\{i\}^\{\(t\+1\)\}\)of the shattered pushforward measure \(Equation[16](https://arxiv.org/html/2609.11253#A1.E16)\)\. To rigorously account for the non\-differentiable decision boundaries∂Ω~k\\partial\\tilde\{\\Omega\}\_\{k\}, we decompose the integral over the latent space into a piecewise sum over the disjoint regionsΩ~k\\tilde\{\\Omega\}\_\{k\}\. Because the boundaries possess a Lebesgue measure of zero, they do not contribute to the integral:

h\(Xi\(t\+1\)\)=−∑k∈𝒵∫Ω~kpi\(t\+1\)\(y\)logpi\(t\+1\)\(y\)dyh\(X\_\{i\}^\{\(t\+1\)\}\)=\-\\sum\_\{k\\in\\mathcal\{Z\}\}\\int\_\{\\tilde\{\\Omega\}\_\{k\}\}p\_\{i\}^\{\(t\+1\)\}\(y\)\\log p\_\{i\}^\{\(t\+1\)\}\(y\)dy
Applying the change of variablesy=ψk​\(x~\)y=\\psi\_\{k\}\(\\tilde\{x\}\)locally within the interior of each regionΩ~k\\tilde\{\\Omega\}\_\{k\}yields:

h⁡\(Xi\(t\+1\)\)=−∑k∈𝒵∫Ω~kp~i\(t\)\(x~\)log\(p~i\(t\)​\(x~\)\|det𝐉ψk​\(x~\)\|\)dx~=h⁡\(X~i\)\+∑k∈𝒵∫Ω~kp~i\(t\)​\(x~\)​log⁡\(\|det𝐉ψk​\(x~\)\|\)​dx~\\begin\{array\}\[\]\{ll\}h\(X\_\{i\}^\{\(t\+1\)\}\)&=\-\\sum\_\{k\\in\\mathcal\{Z\}\}\\int\_\{\\tilde\{\\Omega\}\_\{k\}\}\\tilde\{p\}\_\{i\}^\{\(t\)\}\(\\tilde\{x\}\)\\log\\left\(\\frac\{\\tilde\{p\}\_\{i\}^\{\(t\)\}\(\\tilde\{x\}\)\}\{\|\\det\\mathbf\{J\}\_\{\\psi\_\{k\}\}\(\\tilde\{x\}\)\|\}\\right\)d\\tilde\{x\}\\\\ &=h\(\\tilde\{X\}\_\{i\}\)\+\\sum\_\{k\\in\\mathcal\{Z\}\}\\int\_\{\\tilde\{\\Omega\}\_\{k\}\}\\tilde\{p\}\_\{i\}^\{\(t\)\}\(\\tilde\{x\}\)\\log\\left\(\|\\det\\mathbf\{J\}\_\{\\psi\_\{k\}\}\(\\tilde\{x\}\)\|\\right\)d\\tilde\{x\}\\end\{array\}\(19\)
Because the marginal transformationψk\\psi\_\{k\}operates independently across dimensions, its Jacobian matrix𝐉ψk\\mathbf\{J\}\_\{\\psi\_\{k\}\}is purely diagonal:

det𝐉ψk​\(x~\)=∏d=1Ddd​x~d​ψk,d​\(x~d\)\\det\\mathbf\{J\}\_\{\\psi\_\{k\}\}\(\\tilde\{x\}\)=\\prod\_\{d=1\}^\{D\}\\frac\{d\}\{d\\tilde\{x\}\_\{d\}\}\\psi\_\{k,d\}\(\\tilde\{x\}\_\{d\}\)
Applying the chain rule and the inverse function theorem to the marginal uniformizationψk,d​\(x~d\)=Φ−1​\(∫−∞x~dpk,d\(t\)​\(u\)​𝑑u\)\{\\psi\_\{k,d\}\(\\tilde\{x\}\_\{d\}\)=\\Phi^\{\-1\}\\left\(\\int\_\{\-\\infty\}^\{\\tilde\{x\}\_\{d\}\}p\_\{k,d\}^\{\(t\)\}\(u\)du\\right\)\}\(Equation[12](https://arxiv.org/html/2609.11253#S3.E12)\), we have:

dd​x~d​ψk,d​\(x~d\)=p~k,d\(t\)​\(x~d\)ϕ⁡\(ψk,d​\(x~d\)\)\\frac\{d\}\{d\\tilde\{x\}\_\{d\}\}\\psi\_\{k,d\}\(\\tilde\{x\}\_\{d\}\)=\\frac\{\\tilde\{p\}\_\{k,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\{\\phi\(\\psi\_\{k,d\}\(\\tilde\{x\}\_\{d\}\)\)\}
And thus:

\|det𝐉ψk​\(x~\)\|=∏d=1Dp~k,d\(t\)​\(x~d\)ϕ⁡\(ψk,d​\(x~d\)\)\|\\det\\mathbf\{J\}\_\{\\psi\_\{k\}\}\(\\tilde\{x\}\)\|=\\prod\_\{d=1\}^\{D\}\\frac\{\\tilde\{p\}\_\{k,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\{\\phi\(\\psi\_\{k,d\}\(\\tilde\{x\}\_\{d\}\)\)\}\(20\)
Substituting Equation[20](https://arxiv.org/html/2609.11253#A1.E20)back into the differential entropy expansion \(Equation[19](https://arxiv.org/html/2609.11253#A1.E19)\), and aggregating the piecewise integrals via the empirical predictorη∗​\(x\)\\eta^\{\*\}\(x\), we obtain:

h\(Xi\(t\+1\)\)=h\(X~i\)\+𝔼x~∼X~i\[log∏d=1Dp~η∗​\(x\),d\(t\)\(x~d\)\]−𝔼x~∼X~i\[log∏d=1Dϕ\(ψη∗​\(x\),d\(x~d\)\)\]h\(X\_\{i\}^\{\(t\+1\)\}\)=h\(\\tilde\{X\}\_\{i\}\)\+\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\tilde\{p\}\_\{\\eta^\{\*\}\(x\),d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\\right\]\-\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\phi\(\\psi\_\{\\eta^\{\*\}\(x\),d\}\(\\tilde\{x\}\_\{d\}\)\)\\right\]\(21\)
We formulate the negentropy of the pushforward measureJ⁡\(Xi\(t\+1\)\)J\(X\_\{i\}^\{\(t\+1\)\}\)by substituting Equation[21](https://arxiv.org/html/2609.11253#A1.E21)into Equation[16](https://arxiv.org/html/2609.11253#A1.E16)\. Observing that the standard normal cross\-entropy terms exactly cancel out, the expression simplifies to:

J\(Xi\(t\+1\)\)=−h\(X~i\)−𝔼x~∼X~i\[log∏d=1Dp~η∗​\(x\),d\(t\)\(x~d\)\]J\(X\_\{i\}^\{\(t\+1\)\}\)=\-h\(\\tilde\{X\}\_\{i\}\)\-\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\tilde\{p\}\_\{\\eta^\{\*\}\(x\),d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\\right\]\(22\)
By corollary, the negentropy term corresponding to a perfect oracle routing \(J⁡\(ψi​\(X~i\)\)J\(\\psi\_\{i\}\(\\tilde\{X\}\_\{i\}\)\)\) is recovered by uniformly substitutingη∗​\(x\)=i\\eta^\{\*\}\(x\)=i:

J\(ψi\(X~i\)\)=−h\(X~i\)−𝔼x~∼X~i\[log∏d=1Dp~i,d\(t\)\(x~d\)\]J\(\\psi\_\{i\}\(\\tilde\{X\}\_\{i\}\)\)=\-h\(\\tilde\{X\}\_\{i\}\)\-\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\tilde\{p\}\_\{i,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\\right\]\(23\)
Subtracting Equation[23](https://arxiv.org/html/2609.11253#A1.E23)from Equation[22](https://arxiv.org/html/2609.11253#A1.E22)isolates the routing penaltyEiE\_\{i\}introduced in Equation[18](https://arxiv.org/html/2609.11253#A1.E18):

Ei=𝔼x~∼X~i\[log∏d=1Dp~i,d\(t\)​\(x~d\)p~η∗​\(x\),d\(t\)​\(x~d\)\]E\_\{i\}=\\mathbb\{E\}\_\{\\tilde\{x\}\\sim\\tilde\{X\}\_\{i\}\}\\left\[\\log\\prod\_\{d=1\}^\{D\}\\frac\{\\tilde\{p\}\_\{i,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\{\\tilde\{p\}\_\{\\eta^\{\*\}\(x\),d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\\right\]\(24\)
It needs to be noted thatEiE\_\{i\}is entirely dictated by the misclassified regions, and rewrites as:

Ei=∑k≠i∫Ω~kp~i\(t\)​\(x~\)​log​∏d=1Dp~i,d\(t\)​\(x~d\)p~k,d\(t\)​\(x~d\)​𝑑x~E\_\{i\}=\\sum\_\{k\\neq i\}\\int\_\{\\tilde\{\\Omega\}\_\{k\}\}\\tilde\{p\}\_\{i\}^\{\(t\)\}\(\\tilde\{x\}\)\\log\\prod\_\{d=1\}^\{D\}\\frac\{\\tilde\{p\}\_\{i,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\{\\tilde\{p\}\_\{k,d\}^\{\(t\)\}\(\\tilde\{x\}\_\{d\}\)\}\\,\\mathrm\{d\}\\tilde\{x\}\(25\)
The termEiE\_\{i\}represents the exact entropic penalty incurred by spatial tearing\. It mathematically quantifies the cross\-entropy mismatch caused by evaluating the true marginal densityp~i,d\(t\)\\tilde\{p\}\_\{i,d\}^\{\(t\)\}using the estimators of the mispredicted classp~η∗​\(x\),d\(t\)\\tilde\{p\}\_\{\\eta^\{\*\}\(x\),d\}^\{\(t\)\}\. Because the decision boundaries∂Ω~k\\partial\\tilde\{\\Omega\}\_\{k\}are non\-differentiable, samples routed incorrectly inevitably inject non\-Gaussian artifacts into the pushforward measure\. Consequently, strict monotonic negentropy reduction \(Δ​Ji\>0\\Delta J\_\{i\}\>0\) is not guaranteed\. Noisy routing injects entropy \(Ei\>0E\_\{i\}\>0\), while the subsequent rotation and exact marginalization forcibly remove it \(Δ​Ji∗\>0\\Delta J^\{\*\}\_\{i\}\>0\)\.

Crucially, Equation[25](https://arxiv.org/html/2609.11253#A1.E25)demonstrates that the penaltyEiE\_\{i\}diverges to infinity if a mispredicted marginal densityp~k,d\(t\)\\tilde\{p\}\_\{k,d\}^\{\(t\)\}evaluates to zero over regions where the true densityp~i,d\(t\)\\tilde\{p\}\_\{i,d\}^\{\(t\)\}is strictly positive\. To rigorously constrain this divergence in practice, we enforce a strict minimum thresholdα\>0\\alpha\>0on all empirical marginal density estimators\. By bounding the denominator, the maximum log\-ratio across allDDdimensions becomes strictly finite\. Definingϵi=∑k≠i∫Ω~kp~i\(t\)​\(x~\)​𝑑x~≡𝐏⁡\(η∗​\(X~i\)≠i\)\\epsilon\_\{i\}=\\sum\_\{k\\neq i\}\\int\_\{\\tilde\{\\Omega\}\_\{k\}\}\\tilde\{p\}\_\{i\}^\{\(t\)\}\(\\tilde\{x\}\)\\,\\mathrm\{d\}\\tilde\{x\}\\equiv\\mathbf\{P\}\(\\eta^\{\*\}\(\\tilde\{X\}\_\{i\}\)\\neq i\)as the class\-specific empirical routing error, we can factor it out of the domain of integration, yielding an upper bound of:

Ei≤𝒪⁡\(−ϵi​D​log⁡α\)E\_\{i\}\\leq\\mathcal\{O\}\(\-\\epsilon\_\{i\}D\\log\\alpha\)\(26\)
This thresholding strategy guarantees a strictly finite supremum for the spatial tearing penalty, explicitly tying the worst\-case algorithmic instability to the intrinsic Bayes error of the latent space\.

## Appendix BDatasets

#### GloVe

This dataset is a subset of the 150,000 most frequent word embeddings from the originalGloVecorpus[Pennington et al\. \(2014\)](https://arxiv.org/html/2609.11253#bib.bib24)\. Words are categorized into three discrete concept classes \(male\-biased, female\-biased, and neutral\) based on the magnitude of their projection onto the gender direction\. This direction is formally defined as the principal component of the subspace spanned by gendered word\-pair differences\. The dataset comprises 21,996 embeddings, partitioned into 10,777 for training \(49%\), 4,620 for validation \(21%\), and 6,599 for testing \(30%\)\.

#### Bias in Bios

This is a real\-world benchmark of short biographies scraped from the web, designed specifically to study gender bias in NLP[De\-Arteaga et al\. \(2019\)](https://arxiv.org/html/2609.11253#bib.bib12)\. Each sample is annotated with a binary gender attribute and one of 28 occupation labels\. We utilize the subset curated by[Ravfogel et al\. \(2020\)](https://arxiv.org/html/2609.11253#bib.bib25), which preserves∼\\sim98% of the original corpus\. The dataset exhibits severe historical gender\-occupation correlations\. It contains 399,423 biographies, stratified by occupation into 255,710 training \(64%\), 39,369 validation \(10%\), and 98,344 testing \(25%\) samples\.

#### DIAL

Derived from the DeepMoji corpus[Blodgett et al\. \(2016\)](https://arxiv.org/html/2609.11253#bib.bib7),DIALis a Twitter\-based sentiment classification benchmark\. Each observation includes a binary downstream sentiment label \(happyorsad\) and a binary demographic attribute corresponding to the linguistic dialect: African\-American English \(AAE\) or Standard American English \(SAE\)\. We adopt the version pre\-processed by[Chowdhury et al\. \(2023\)](https://arxiv.org/html/2609.11253#bib.bib10), which guarantees perfect balance across both race and sentiment labels\. The dataset totals 175,996 samples, split into 160,000 for training \(91%\), 8,000 for validation \(4\.5%\), and 7,996 for testing \(4\.5%\)\.

#### Jigsaw

Based on the Jigsaw Toxicity Classification benchmark, this dataset is used for the downstream task of binary toxicity detection\. We target religion as the sensitive concept, operationalized as a discrete categorical variable with five labels:Buddhist,Christian,Hindu,Jewish, andMuslim\. The corpus comprises 96,492 samples, divided into 87,434 for training \(91%\) and 9,058 for testing \(9%\)\.

## Appendix CTraining

### C\.1Hardware and Environment

All experiments are conducted on an NVIDIA GeForce RTX 2080 Ti GPU with 11GB of VRAM and an Intel\(R\) Core\(TM\) i9\-9900K CPU, using PyTorch 2\.8\.0 and Python 3\.12\.9\. We employ CUDA 12\.6 for accelerated computations\.

### C\.2MUtE

We selectedTTvia early stopping, halting the procedure when the validation probe accuracy plateaued at the majority\-class baseline, resulting in:T=100T=100forGloVe,T=70T=70forBias in Bios,T=150T=150forDIALandJigsaw\.

### C\.3Other models settings

#### LEACE\.

#### KRaM and FaRM\.

We reimplement both KRaM[Chowdhury et al\. \(2023\)](https://arxiv.org/html/2609.11253#bib.bib10)and FaRM[Chowdhury and Chaturvedi \(2022\)](https://arxiv.org/html/2609.11253#bib.bib9)following a standardized training protocol\. For both architectures, the hidden dimension at each layer is strictly constrained to match the input feature dimension\. Network depth is task\-dependent: we utilize 4 layers forGloVe,Bias in Bios, andJigsaw, and 7 layers forDIAL\. The networks are optimized over 50 epochs with a batch size of 512, utilizing a learning rate of10−310^\{\-3\}and a weight decay coefficient of10−510^\{\-5\}\. For the KRaM objective specifically, the regularization weighting hyperparameter is set toλ=0\.7\\lambda=0\.7\.

#### L¯\\overline\{\\mathrm\{\\textbf\{L\}\}\}EOPARD\.

We reimplemented theL¯\\overline\{\\mathrm\{L\}\}EOPARD framework[Saillenfest and Lemberger \(2025\)](https://arxiv.org/html/2609.11253#bib.bib29)\. For theGloVe,Bias in bios, andDIALdatasets, we adopted the exact hyperparameter configurations recommended by the original authors\. Lacking explicit guidelines forJigsaw, we applied theDIALconfiguration to this dataset\. Cascaded training was employed across all settings\. Specifically, models were trained for10001000epochs onGloVe,100100epochs onBias in bios, and200200epochs on bothDIALandJigsaw, utilizing batch sizes of1077710777,81928192, and20482048, respectively\. The regularization parameterγ\\gammawas set to200200forGloVeand100100for all other datasets\. The initial learning rate was configured to1\.0×10−31\.0\\times 10^\{\-3\}forGloVeand5\.0×10−45\.0\\times 10^\{\-4\}for the remaining datasets, subject to a step decay factor of0\.10\.1at the training midpoint\.

#### TaCo\.

We reimplemented the TaCo framework[Jourdan et al\. \(2024\)](https://arxiv.org/html/2609.11253#bib.bib18)\. Standard TaCo operates as a constrained nonlinear erasure method, optimizing post\-erasure representations by ranking removal directions according to their joint importance to both the target concept and a predefined downstream label\. For a fair evaluation against unconstrained baselines, we ablated this downstream dependency\. Specifically, we adapted TaCo to rank directions based exclusively on their relevance to the target concept, quantified via variance\-based sensitivity analysis using Sobol indices\. Consistent with[Jourdan et al\. \(2024\)](https://arxiv.org/html/2609.11253#bib.bib18), we initially apply PCA to project the representations onto a100100\-dimensional subspace\. To trace the Pareto front \(Figure[3](https://arxiv.org/html/2609.11253#S4.F3)\), we sweep the number of filtered dimensions from55to9595by step of55\.

### C\.4Evaluation Protocol

To evaluate representation quality, both the probing classifiers and the downstream task models are implemented viascikit\-learn’sMLPClassifier\. The networks are optimized using a constant learning rate of10−410^\{\-4\}for a maximum of 20 training epochs, which empirically proved sufficient for convergence\. To ensure statistical reliability, all reported accuracy and fairness metrics are averaged across five independent experimental runs\.

## Appendix DFairness Metrics

TPRRMS\\mathrm\{TPR\}^\{\\mathrm\{RMS\}\}is the root mean square \(RMS\) of the sum of the bias quantified by computing the difference \(Gap\) in the true positive rate \(TPR\) of the classifier between individuals with different concept class labels\. Formally, for a binary concept and downstream labelsYYsampled from the set of downstream labels𝒴\\mathcal\{Y\}:

TPR0,y=p⁡\(Y^=y\|Z=0,Y=y\)Gapy=TPR1,y−TPR0,yTPRRMS=1\|𝒴\|​∑y∈𝒴\(Gapy\)2\\begin\{split\}\\mathrm\{TPR\}\_\{0,y\}=p\(\\hat\{Y\}=y\|Z=0,Y=y\)\\\\ \\mathrm\{Gap\}\_\{y\}=\\mathrm\{TPR\}\_\{1,y\}\-\\mathrm\{TPR\}\_\{0,y\}\\\\ \\mathrm\{TPR\}^\{\\mathrm\{RMS\}\}=\\sqrt\{\\frac\{1\}\{\|\\mathcal\{Y\}\|\}\\sum\_\{y\\in\\mathcal\{Y\}\}\(\\mathrm\{Gap\}\_\{y\}\)^\{2\}\}\\end\{split\}\(27\)
Intuitively, the true\-positive\-rate of a “fair” classifier should not be sensitive to the values of the protected attributes\.

For completeness, we also report demographic parity which measures the difference in prediction w\.r\.t\. to a protected attribute and is achieved when the probability of a certain prediction is not dependent on sensitive group membership:

DP=∑y∈𝒴\|p⁡\(Y^=y\|Z=0\)−p⁡\(Y^=y\|Z=1\)\|\\mathrm\{DP\}=\\sum\_\{y\\in\\mathcal\{Y\}\}\|p\(\\hat\{Y\}=y\|Z=0\)\-p\(\\hat\{Y\}=y\|Z=1\)\|\\\\\(28\)
Table 6:Empirical evaluation of the MLP surrogate’s fidelity relative to the exactMUtEmapping\. Inference time indicates the total duration required to map the respective test sets\. Downstream performance and fairness metrics are evaluated using classifiers trained on the exactMUtErepresentations but tested on representations generated by the designated model at inference time\.
## Appendix EDistillation of the MUtE Mapping

As a preliminary proof\-of\-concept for offline distillation, we approximate a pre\-trained exactMUtEmapping using a Multi\-Layer Perceptron \(MLP\) surrogate\. The surrogate pipeline comprises an initial Principal Component Analysis \(PCA\) whitening step \(fitted on the training distribution\) followed by a 4\-layer MLP\. To ensure sufficient capacity, the hidden dimension is set to2​d2d, whereddrepresents the input feature dimension\. Intermediate layers consist of a linear transformation, Layer Normalization, and atanh\\tanhactivation, culminating in a linear output layer\.

We train the surrogate model to minimize the Mean Squared Error \(MSE\) between its outputs and the exactMUtErepresentations\. Optimization is performed using AdamW with a learning rate of10−310^\{\-3\}and a batch size of2,0482,048for a maximum of1,5001,500epochs, holding out2,0002,000samples for validation monitoring\.

As detailed in Table[6](https://arxiv.org/html/2609.11253#A4.T6), the MLP surrogate strictly preserves both the predictive utility \(aya\_\{y\}\) and the fairness guarantees \(TPRRMS\\mathrm\{TPR\}^\{\\mathrm\{RMS\}\}, DP,aza\_\{z\}\) of the exactMUtEmapping\. Crucially, the surrogate accelerates inference by at least an order of magnitude\. This empirically validates that a lightweight feed\-forward network can successfully capture the underlying geometric manifold of the optimal erasure function, providing an efficient alternative for latency\-sensitive downstream deployments\.