Efficient bias mitigation in T2I diffusion models using Concept Graphs

# Efficient bias mitigation in T2I diffusion models using Concept Graphs
Source: [https://arxiv.org/html/2607.03397](https://arxiv.org/html/2607.03397)
Mansi Department of Computing Imperial College London m\.\-24@imperial\.ac\.uk &Avinash Kori Department of Computing Imperial College London a\.kori21@imperial\.ac\.uk &Francesco Leofante Department of Computing Imperial College London f\.leofante@imperial\.ac\.uk

###### Abstract

Text\-to\-Image diffusion models often propagate harmful bias inherited from the training data\. Existing bias mitigation techniques typically intervene only at the text encoder or provide inference\-time guidance, often leading to generations that collapse into*semantically incoherent outputs*\. To address these limitations, we introduceCO\-ALIGN\(Concept Ontology Alignment\), a novel bias mitigation approach based on concept\-graph alignment that operates on the model’s*internal concept ontology*\. By aligning concepts within the text encoder and denoiser,CO\-ALIGNachieves substantial bias reduction while preserving generative integrity\. We demonstrate the effectiveness of concept\-graph alignment across three paradigms: text\-encoders, denoisers and joint text\-denoiser ontology alignment\.CO\-ALIGNoutperforms the state of the art, improving fairness by30%30\\%,Δ​F​I​D=11\.4\\Delta FID=11\.4in image quality,2\.8%2\.8\\%in image fidelity, all while reducing semantically incoherent outputs by88%88\\%\. Beyond bias mitigation, we show that CO\-ALIGN benefits other downstream tasks as well\. In particular, our experiments demonstrate that better\-aligned internal ontologies enhance concept unlearning robustness across multiple unlearning techniques\.

## 1Introduction

Text\-to\-image \(T2I\) diffusion models have achieved remarkable generative capability, yet they systematically reproduce and amplify the demographic biases present in their training corpora\(Schuhmannet al\.,[2022b](https://arxiv.org/html/2607.03397#bib.bib1); Bianchiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib2)\)\. These biases are not just superficial stylistic artifacts, but structural properties of the model’s learned concept associations\. Once embedded in the weights, they are reproduced consistently at every generation\. To address this, current bias mitigation methods intervene at one of two levels: the*text encoder*, via text embedding fine\-tuning\(Shenet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib23)\)or inference\-time steering\(Friedrichet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib9); Chuanget al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib11)\); or the*denoiser*, via cross\-attention weight editing\(Orgadet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib13); Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\)or latent space manipulation\(Pariharet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib16); Liet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib17); Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\)\. Both paradigms share a common and consequential assumption: that the text encoder and denoiser can be treated as*independent*correction targets, and that fixing one is sufficient to correct the generated output\.

The problem with existing approaches\.The denoiser usually adapts to the given text\-encoder, or in the case of end\-to\-end training, both text encoder and the denoiser are jointly updated, without explicit*disentanglement*guarantees: the denoiser’s cross\-attention heads learn to expect the*specific biased embedding geometry*produced by the original text encoder, and have organized their internal concept representations around producing biased visual outputs\. When only the text encoder is edited, the denoiser receives an out\-of\-distribution conditioning signal, which it often cannot interpret coherently, resulting in*semantically incoherent*generations that depict neither the target concept nor any meaningful content\. Conversely, when only the denoiser is edited, the text encoder continues to route underspecified prompts \(e\.g\.,“a nurse”, where the gender is not mentioned\) toward biased attribute embeddings, partially negating the denoiser\-side correction\.

![Refer to caption](https://arxiv.org/html/2607.03397v1/x1.png)Figure 1:Existing bias mitigation methods that intervene on only the text encoder \(left\) or denoiser \(centre\) produce either semantically incoherent outputs or gender\-biased generations for the prompt“Nurse”\.CO\-ALIGN\(right\) jointly aligns the concept graphs of both components toward a target topology, restoring both coherence and demographic balance\. CG representes the concept graphs\.Our approach\.We propose to mitigate this by aligning the model’s*internal concept ontology*directly\. We proposeCO\-ALIGN\(ConceptOntologyALIGNment\), a framework that extracts and edits the structured relational graph of*concept*representations distributed across both the text encoder’s embedding space and the denoiser’s cross\-attention layers\. The key insight underlyingCO\-ALIGNis that bias is not a property of individual concept representations, but of the*relationships between them*\. In the original model, biased concepts share disproportionately more neural substrate with their stereotypically associated attributes than with their counter\-stereotypical ones, like a nurse if majorly attributed to a ‘female’ nurse over a ‘male’ nurse\. This asymmetry is observable in the extracted concept graph via asymmetric edge weights\.CO\-ALIGNaddresses this by aligning the concept graph toward a*target topology*, bringing under\-represented concept associations closer to the target concept, jointly in both the text encoder and the denoiser, consequently restoring balanced representation in the final generation\. As shown in our experiments, this joint editing enables alignment of the geometric representations in the text encoder and the denoiser, leading to faithful and unbiased generation\.

Emergent property inCO\-ALIGN\.Beyond our primary debiasing objective, we find that aligning concept graphs throughCO\-ALIGNcauses*semantic neighbourhood propagation*, i\.e\.*aligning one concept pulls its semantic neighbourhood along with it\.*This effect is analogous to label propagation in graph\-based semi\-supervised learning\(Zhouet al\.,[2003](https://arxiv.org/html/2607.03397#bib.bib43); Zhuet al\.,[2003](https://arxiv.org/html/2607.03397#bib.bib44); Iscenet al\.,[2019](https://arxiv.org/html/2607.03397#bib.bib45)\), where supervision on a small set of labelled nodes propagates to adjacent unlabeled nodes via the graph smoothness assumption\(Belkinet al\.,[2006](https://arxiv.org/html/2607.03397#bib.bib46)\)\. We leverage this property, along with the locality of concept erasure property of unlearning techniques \(Buiet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib42)\)\), for a second downstream application of robust unlearning\. By aligning a small set of adversarial bypass concepts toward the unlearning target, before applying any unlearning technique,CO\-ALIGNsignificantly improves the robustness of post\-hoc unlearning techniques against adversarial prompts\.

Summary of contributions\.Our main contributions can be summarized as follows\. ❶ We show that current single\-component debiasing techniques trade off bias for inconsistent generation and residual bias\. ❷ Motivated by this analysis, we proposeCO\-ALIGN, a bias mitigation framework that extracts a concept\-level graph from both the text encoder and the denoiser, and aligns it toward a target topology jointly across both components\. ❸ We demonstrate thatCO\-ALIGNoutperforms existing state of the art debiasing techniques in producing class balanced and coherent outputs while preserving the generative quality of the output\. ❹ We empirically characterize a neighbourhood propagation effect and leverage it for significantly improving the adversarial robustness of existing unlearning techniques usingCO\-ALIGN\.

## 2Inherent Biases in T2I Diffusion Models

Dataset biases and their implications\.Alike other generative models, biases in T2I diffusion models often originate from the large\-scale web\-scraped corpora used for training\. LAION\-5B\(Schuhmannet al\.,[2022b](https://arxiv.org/html/2607.03397#bib.bib1)\), used for training Stable Diffusion, mirrors societal inequalities at internet scale across profession\(Bianchiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib2)\), race\(Choet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib3); Luccioniet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib33)\)and culture\(Naik and Nushi,[2023](https://arxiv.org/html/2607.03397#bib.bib4)\)\. These biases have shown to be*amplified*beyond the degree present in training data both during model training\(Seshadriet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib5); Perera and Patel,[2023a](https://arxiv.org/html/2607.03397#bib.bib6)\)and inference\(Rooset al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib8)\)\. Large scale deployment of these models in both human operated and automated systems have shown detrimental societal consequences of reinforcement of stereotypes\(Bianchiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib2)\), across gender, age, race, and geography simultaneously\(Naik and Nushi,[2023](https://arxiv.org/html/2607.03397#bib.bib4)\)\.

Bias mitigation techniques\.Existing debiasing techniques can be broadly classified in two groups, depending on where in the generation pipeline they intervene, as summarised in Table[1](https://arxiv.org/html/2607.03397#S2.T1)\.

Text encoder debiasingmethods intervene solely at the text encoder\. Shen et al\.\(Shenet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib23)\)fine\-tune the text encoder directly using a distributional alignment loss on generated images\. Fair Mapping\(Liet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib40)\)learns a lightweight linear remap over conditioning embeddings to project them into a debiased subspace\. Chuang et al\.\(Chuanget al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib11)\)achieve a similar effect training\-free, via a calibrated projection matrix that removes biased directions from text embeddings at inference time\. Fair Diffusion\(Friedrichet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib9)\)via SEGA\(Bracket al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib10)\), FairGen\(Kanget al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib39)\), and Kim et al\.\(Kimet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib12)\)take a softer approach, steering the conditioning signal or noise initialisation at inference without any weight modification\. While efficient, all text\-encoder\-only methods leave the denoiser’s internal concept organization and the visual biases encoded\.

Denoiser debiasingmethods target the denoiser’s internal representations\. TIME\(Orgadet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib13)\)and UCE\(Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\)edit cross\-attention key\-value projections in closed form to reroute underspecified prompts toward target attribute embeddings\. Asyrp\(Kwonet al\.,[2023a](https://arxiv.org/html/2607.03397#bib.bib15)\)establishes the UNet bottleneck \(h\-space\) as a semantically linear space for controllable editing; building on this, Balancing Act\(Pariharet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib16)\)trains a lightweight predictor on h\-space features, Li et al\.\(Liet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib17)\)discover fairness\-sensitive directions without external classifiers, Vardhana et al\.\(Vardhanaet al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib18)\)steer denoising toward a uniform attribute distribution in an unsupervised manner, and SCALEX\(Zenget al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib22)\)maps conceptual structure via prompt\-aligned latent directions\. DIFFLENS\(Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\)takes a mechanistic approach, using sparse autoencoders to identify and suppress neuron\-level dimensions responsible for bias; BiasMap\(Chakrabortyet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib20)\), EFA\(Parket al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib37)\), and Yasser et al\.\(Yasseret al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib21)\)further reveal that many such corrections reduce distributional gaps without disentangling the underlying concept coupling\. MAS\(Zhouet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib38)\)addresses the related problem of association\-engendered stereotypes from co\-generation of multiple concepts\. While these methods edit the denoiser, they are applied independently of the text encoder, leaving the biased embedding geometry of underspecified prompts uncorrected\.

Table 1:Structured comparison of bias mitigation methods across six dimensions, encompassing training dynamics \(columns \[1–3\]\), editing region \(columns \[4–5\]\) and scalability \(column \[6\]\)\. CO\-ALIGN edits both the text encoder and the denoiser while supporting multi\-concept joint editing with no added inference cost\.MethodWeightMod\.\[1\]TrainingFree\[2\]Infer\.Cost\[3\]TEEdit\[4\]Den\.Edit\[5\]Multi\-Concept\[6\]Fair DiffusionFriedrichet al\.\([2023](https://arxiv.org/html/2607.03397#bib.bib9)\)✗✓✓✗✗✗Debiasing VLMsChuanget al\.\([2023](https://arxiv.org/html/2607.03397#bib.bib11)\)✗✓✓✓✗✗FairGenKanget al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib39)\)✗✓✓✗✗✗Kim et al\.Kimet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib12)\)✗✓✓✗✗✗EFAParket al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib37)\)✗✓✓✗✓✗H\-DistributionPariharet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib16)\)✗✓✓✗✓✗Latent DirectionLiet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib17)\)✗✓✓✗✓✗SelfDebiasVardhanaet al\.\([2026](https://arxiv.org/html/2607.03397#bib.bib18)\)✗✓✓✗✓✗SCALEXZenget al\.\([2026](https://arxiv.org/html/2607.03397#bib.bib22)\)✗✓✓✗✓✗DiffLensShiet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib19)\)✗✓✓✗✓✓BiasMapChakrabortyet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib20)\)✗✓✓✗✓✗AsyrpKwonet al\.\([2023a](https://arxiv.org/html/2607.03397#bib.bib15)\)✗✓✓✗✓✗Fair MappingLiet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib40)\)✓✗✗✓✗✗TIMEOrgadet al\.\([2023](https://arxiv.org/html/2607.03397#bib.bib13)\)✓✓✗✗✓✗UCEGandikotaet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib14)\)✓✓✗✗✓✓FinetuningShenet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib23)\)✓✗✗✓✗✗MASZhouet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib38)\)✓✗✗✓✗✓CO\-ALIGN \(Ours\)✓✗✗✓✓✓

## 3Understanding bias

As discussed in section[A](https://arxiv.org/html/2607.03397#A1), the major root cause for biased behavior arises from the inherent bias in the training dataset\. Figure[6](https://arxiv.org/html/2607.03397#A1.F6)\(in appendix\) shows the distributional bias across occupations for gender, race and age in the LAION 2B datasetSchuhmannet al\.\([2022a](https://arxiv.org/html/2607.03397#bib.bib53)\)\(with FairFace classifier\(Kärkkäinen and Joo,[2021](https://arxiv.org/html/2607.03397#bib.bib52)\)\) used for training Stable Diffusion 1\.5\. This bias is then amplified by the model during the training phase due to the tendency of the reverse diffusion process to navigate towards dominant modesSeshadriet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib5)\); Perera and Patel \([2023a](https://arxiv.org/html/2607.03397#bib.bib6),[b](https://arxiv.org/html/2607.03397#bib.bib54)\), resulting in the generation of biased content\.

![Refer to caption](https://arxiv.org/html/2607.03397v1/x2.png)Figure 2:\(a\) the cosine similarity between the text encoder’s embeddings of concepts of Nurse, Female Nurse and Male Nurse; and \(b\) Figure comparing the class imbalance % \(x\-axis: lower is better\) against Incoherence rate \(y\-axis: fraction of generations unclassifiable as the target, lower is better\) for five debiasing methods and the base model\.Is it a Query Understanding or Model Interpretation Problem?A fundamental question that divides the bias mitigation literature is whether the observed generative bias is primarily a failure of*query understanding*,*i\.e\.*, the text encoder misinterprets the input prompt, or a failure of*model interpretation**i\.e\.*, the denoiser’s internal visual concept representations are themselves skewed, and would produce biased outputs regardless of prompt comprehension\. Several observations from the literature suggest that both components are affected, but in structurally distinct ways\. On the text encoder, underspecified prompts are implicitly enriched with demographic associations\(Orgadet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib13)\), and removing biased directions from text embeddings alone produces measurably fairer outputs\(Chuanget al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib11)\)\. Figure[2](https://arxiv.org/html/2607.03397#S3.F2)\(a\) supports this argument, where we see that the cosine similarity between the text encoder’s embeddings ofNurse,Female Nurse, andMale Nursereveals that the model’s query understanding ofNurseis inherently skewed toward the feminine\.

On the denoiser side, specific neuron\-level dimensions in the denoiser’s h\-space are independently responsible for demographic bias\(Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\), and gender bias is present in the cross\-attention layers regardless of the text conditioning signal\(Wuet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib7)\)\. While an analogous diagnostic does not yet exist for the denoiser, we hypothesize that a similar relational concept graph extracted from its cross\-attention layers would reveal a structurally similar pattern\. We later confirm this hypothesis empirically in Section[4](https://arxiv.org/html/2607.03397#S4), whereCO\-ALIGNextracts exactly such a graph and shows that the denoiser’s internal concept topology mirrors the text encoder’s bias\.

Hypothesis: joint alignment is necessary\.Single\-component interventions are suboptimal because the text encoder and denoiser share an entangled geometric representation: correcting one without the other disrupts this alignment, degrading generation coherence even as class imbalance improves\. Figure[2](https://arxiv.org/html/2607.03397#S3.F2)\(b\) illustrates this trade\-off: text\-encoder\-only methods \(Debias\-VL\(Chuanget al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib11)\), Finetuning\(Shenet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib23)\)\) and denoiser\-only methods \(UCE\(Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\), Latent Directions\(Liet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib17)\)\) all reduce class imbalance at the cost of a sharp increase in semantically incoherent generations\. We therefore hypothesize that effective debiasing requires*joint*alignment of both components, preserving the geometric harmony between them\. We later confirm this hypothesis empirically:CO\-ALIGNachieves lower class imbalance than all single\-component baselines while maintaining generation coherence comparable to the unedited base model\.

## 4Extraction of a Concept Graph

Background on Concept Neurons\.Prior work\(Basuet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib49); Fanet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib48); Liuet al\.,[2023b](https://arxiv.org/html/2607.03397#bib.bib50)\)has established that information about a specific concept is concentrated in a sparse subset of parameters within the CA projection matrices, termed*concept neurons*\. Following TRUST\(Mansiet al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib51)\), we define a parameterθcu\\theta\_\{c\_\{u\}\}as a concept neuron for conceptcuc\_\{u\}if perturbing its value produces a measurable change in the model’s alignment tocuc\_\{u\}, quantified via CLIP score:

ℒcu\\displaystyle\\mathcal\{L\}\_\{c\_\{u\}\}=CLIPScore​\(Icu,cu\),\\displaystyle=\\text\{CLIPScore\}\(I\_\{c\_\{u\}\},\\,c\_\{u\}\),\(1\)ℳr​\(cu\)\\displaystyle\\mathcal\{M\}\_\{r\}\(c\_\{u\}\)=η​\(𝔼​\[\|∇θ=θ0ℒcu\|\]\>γ\)r,\\displaystyle=\\eta\\\!\\left\(\\mathbb\{E\}\\\!\\left\[\|\\nabla\_\{\\theta=\\theta\_\{0\}\}\\mathcal\{L\}\_\{c\_\{u\}\}\|\\right\]\>\\gamma\\right\)^\{r\},\(2\)whereIcuI\_\{c\_\{u\}\}is an image generated conditioned oncuc\_\{u\},η​\(⋅\)\\eta\(\\cdot\)is an element\-wise indicator, andγ=ξ⋅σG\+μG\\gamma=\\xi\\cdot\\sigma\_\{G\}\+\\mu\_\{G\}is an adaptive threshold over the gradient matrixG=\|∇θ=θ0ℒcu\|rG=\|\\nabla\_\{\\theta=\\theta\_\{0\}\}\\mathcal\{L\}\_\{c\_\{u\}\}\|^\{r\}\.CO\-ALIGNuses concept neurons not for unlearning, but to construct a concept\-level knowledge graph: nodes represent the cumulative strength of concept neuron populations per concept, and edges represent the shared neuron strength between concepts\.

![Refer to caption](https://arxiv.org/html/2607.03397v1/x3.png)Figure 3:The Figure showsCO\-ALIGN’s three staged finetuning pipeline\. In the concept graphs the dotted nodes represent Target Concepts and solid nodes represent the Anchor Concepts\.Let𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}be a concept vocabulary\. We represent the concept graph as a weighted, symmetric adjacency matrix𝐀∈ℝn×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{n\\times n\}, whereAi​jA\_\{ij\}quantifies the shared neural substrate betweencic\_\{i\}andcjc\_\{j\}\. We define such graphs separately for the text encoder \(§[3](https://arxiv.org/html/2607.03397#S4.F3)\) and denoiser \(§[3](https://arxiv.org/html/2607.03397#S4.F3)\), then jointly align them in §[5](https://arxiv.org/html/2607.03397#S5)\.

Text Encoder\.The concept representation ofcic\_\{i\}is the mean\-pooled last hidden state of the text encoderfϕf\_\{\\phi\}:

𝐞i=1Li​∑ℓ=1Lifϕ​\(ci\)ℓ∈ℝd\.\\mathbf\{e\}\_\{i\}=\\frac\{1\}\{L\_\{i\}\}\\sum\_\{\\ell=1\}^\{L\_\{i\}\}f\_\{\\phi\}\(c\_\{i\}\)\_\{\\ell\}\\;\\in\\;\\mathbb\{R\}^\{d\}\.\(3\)Thetext\-encoder concept graph𝒢TE=\(𝒞,𝐀TE\)\\mathcal\{G\}^\{\\mathrm\{TE\}\}=\(\\mathcal\{C\},\\,\\mathbf\{A\}^\{\\mathrm\{TE\}\}\)is defined by pairwise cosine similarity ofℓ2\\ell\_\{2\}\-normalised embeddings:

Ai​jTE=𝐞i⊤​𝐞j‖𝐞i‖​‖𝐞j‖=𝐞^i⊤​𝐞^j,𝐀TE=E^​E^⊤,A^\{\\mathrm\{TE\}\}\_\{ij\}=\\frac\{\\mathbf\{e\}\_\{i\}^\{\\top\}\\mathbf\{e\}\_\{j\}\}\{\\\|\\mathbf\{e\}\_\{i\}\\\|\\,\\\|\\mathbf\{e\}\_\{j\}\\\|\}=\\hat\{\\mathbf\{e\}\}\_\{i\}^\{\\top\}\\hat\{\\mathbf\{e\}\}\_\{j\},\\qquad\\mathbf\{A\}^\{\\mathrm\{TE\}\}=\\hat\{E\}\\,\\hat\{E\}^\{\\top\},\(4\)whereE^=\[𝐞^1,…,𝐞^n\]⊤∈ℝn×d\\hat\{E\}=\[\\hat\{\\mathbf\{e\}\}\_\{1\},\\ldots,\\hat\{\\mathbf\{e\}\}\_\{n\}\]^\{\\top\}\\in\\mathbb\{R\}^\{n\\times d\}\. Off\-diagonal entries lie in\[−1,1\]\[\-1,1\]and capture directional alignment of concept embeddings\.

Denoiser\.Unlike the text encoder, the denoiser has no single embedding per concept; concept binding is distributed across the cross\-attention projection matricesWK,WQ,WVW\_\{K\},W\_\{Q\},W\_\{V\}\. We represent each concept’s position in the denoiser’s ontology by itsconcept neuron fingerprint— the masked gradient profile over cross\-attention weights\.

For conceptcuc\_\{u\}and projection typer∈\{k,q,v\}r\\in\\\{k,q,v\\\}, the per\-head importance of headhhin layerℓ\\ellis:

Grℓ,h​\(cu\)=𝔼​\[\|∇Wrℓ,hℒcu\|\]mean\.G^\{\\ell,h\}\_\{r\}\(c\_\{u\}\)=\\mathbb\{E\}\\\!\\left\[\\,\\bigl\|\\nabla\_\{W^\{\\ell,h\}\_\{r\}\}\\mathcal\{L\}\_\{c\_\{u\}\}\\bigr\|\\,\\right\]\_\{\\text\{mean\}\}\.\(5\)Stacking over allLLlayers andHHheads yields𝐆r​\(cu\)∈ℝL​H\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\\in\\mathbb\{R\}^\{LH\}\. The masked fingerprint is:

ℳr​\(cu\)\\displaystyle\\mathcal\{M\}\_\{r\}\(c\_\{u\}\)=η​\(𝐆r​\(cu\)\>γ\),γ=ξ⋅σ​\(𝐆r​\(cu\)\)\+μ​\(𝐆r​\(cu\)\),\\displaystyle=\\eta\\\!\\left\(\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\>\\gamma\\right\),\\quad\\gamma=\\xi\\cdot\\sigma\(\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\)\+\\mu\(\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\),\(6\)𝐠r​\(cu\)\\displaystyle\\mathbf\{g\}\_\{r\}\(c\_\{u\}\)=𝐆r​\(cu\)⊙ℳr​\(cu\)∈ℝL​H\.\\displaystyle=\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\\odot\\mathcal\{M\}\_\{r\}\(c\_\{u\}\)\\in\\mathbb\{R\}^\{LH\}\.\(7\)Thedenoiser concept graph𝒢𝒰=\(𝒞,𝐀𝒰\)\\mathcal\{G\}^\{\\mathcal\{U\}\}=\(\\mathcal\{C\},\\,\\mathbf\{A\}^\{\\mathcal\{U\}\}\)is then:

Ai​j𝒰=13​∑r∈\{k,q,v\}cos⁡\(𝐠r​\(ci\),𝐠r​\(cj\)\),A^\{\\mathcal\{U\}\}\_\{ij\}=\\frac\{1\}\{3\}\\sum\_\{r\\in\\\{k,\\,q,\\,v\\\}\}\\cos\\\!\\left\(\\mathbf\{g\}\_\{r\}\(c\_\{i\}\),\\;\\mathbf\{g\}\_\{r\}\(c\_\{j\}\)\\right\),\(8\)where a highAi​j𝒰A^\{\\mathcal\{U\}\}\_\{ij\}indicates thatcic\_\{i\}andcjc\_\{j\}share disproportionate neural substrate in the denoiser’s cross\-attention layers\.

## 5Concept Graphs for Bias Mitigation

Concept Graph Registration\.The target graph𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}encodes the desired concept topology\. The concept vocabulary is partitioned into atarget conceptc∗c^\{\*\}\(e\.g\. “Nurse”\), free to adapt during training, andanchor concepts𝒞S\\mathcal\{C\}\_\{S\}\(e\.g\. \{“Male Nurse”, “Female Nurse”\}\), held fixed viaℒanchor\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\. The registration encodes equal proximity ofc∗c^\{\*\}to all attribute concepts:

Ac∗,cGT=s∗,∀c∈𝒞S,A^\{\\mathrm\{GT\}\}\_\{c^\{\*\},\\,c\}=s^\{\*\},\\quad\\forall\\,c\\in\\mathcal\{C\}\_\{S\},\(9\)wheres∗∈\[0,1\]s^\{\*\}\\in\[0,1\]is the target similarity \(typicallys∗=1\.0s^\{\*\}=1\.0\)\. The supervised row set is𝒮=\{c∗\}\\mathcal\{S\}=\\\{c^\{\*\}\\\}, soℒrank\\mathcal\{L\}\_\{\\mathrm\{rank\}\}propagates gradients only through the target concept’s row\. Alignment is applied sequentially, first to the text encoder, then to the denoiser recalibrated to the updated geometry, as illustrated in Figure[4](https://arxiv.org/html/2607.03397#S6.F4)\.

Now that we have the target graph𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}, we define a specialized training objective to achieve its registration with𝐀TE\\mathbf\{A\}^\{\\mathrm\{TE\}\}and𝐀𝒰\\mathbf\{A\}^\{\\mathcal\{U\}\}\.

Training Objective\.Let𝐀\\mathbf\{A\}denote either𝐀TE\\mathbf\{A\}^\{\\mathrm\{TE\}\}or𝐀𝒰\\mathbf\{A\}^\{\\mathcal\{U\}\}, and let𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}be a target adjacency matrix encoding the desired concept topology \(§[5](https://arxiv.org/html/2607.03397#S5)\)\. CO\-ALIGN aligns𝐀\\mathbf\{A\}toward𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}via a differentiable ranking loss that operates row\-wise over a supervised subset𝒮⊆𝒞\\mathcal\{S\}\\subseteq\\mathcal\{C\}\.Differentiable ranking loss\.For each conceptci∈𝒮c\_\{i\}\\in\\mathcal\{S\}, the adjacency matrix𝐀\\mathbf\{A\}induces a ranking of all concepts by proximity tocic\_\{i\}\. We encode this viasoft ranksusing differentiable sigmoid pairwise comparisons, and align them tohard rank targetsr¯i​jGT\\bar\{r\}^\{\\mathrm\{GT\}\}\_\{ij\}derived from𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}:

r~i​j\\displaystyle\\tilde\{r\}\_\{ij\}=1n​\(1\+∑k=1nσ​\(Ai​k−Ai​jτ\)\),r~i​j∈\(1n,1\],\\displaystyle=\\frac\{1\}\{n\}\\\!\\left\(1\+\\sum\_\{k=1\}^\{n\}\\sigma\\\!\\\!\\left\(\\frac\{A\_\{ik\}\-A\_\{ij\}\}\{\\tau\}\\right\)\\\!\\right\),\\quad\\tilde\{r\}\_\{ij\}\\in\\left\(\\tfrac\{1\}\{n\},\\;1\\right\],\(10\)r¯i​jGT\\displaystyle\\bar\{r\}^\{\\mathrm\{GT\}\}\_\{ij\}=1n⋅avgrankk⁡\(Ai​jGT\|row​i\),\\displaystyle=\\frac\{1\}\{n\}\\cdot\\operatorname\{avgrank\}\_\{k\}\\\!\\\!\\left\(A^\{\\mathrm\{GT\}\}\_\{ij\}\\;\\big\|\\;\\text\{row\}\\ i\\right\),\(11\)ℒrank\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{rank\}\}=1\|𝒮\|⋅n​∑i∈𝒮∑j=1n\(r~i​j−r¯i​jGT\)2,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{S\}\|\\cdot n\}\\sum\_\{i\\in\\mathcal\{S\}\}\\;\\sum\_\{j=1\}^\{n\}\\\!\\left\(\\tilde\{r\}\_\{ij\}\-\\bar\{r\}^\{\\mathrm\{GT\}\}\_\{ij\}\\right\)^\{\\\!2\},\(12\)whereτ\>0\\tau\>0controls sharpness and largerr~i​j\\tilde\{r\}\_\{ij\}indicates lower similarity rank relative tocic\_\{i\}\.

Static anchor loss\.A subset𝒞S⊂𝒞\\mathcal\{C\}\_\{S\}\\subset\\mathcal\{C\}ofanchor conceptsretain their pre\-alignment representations\. For the text encoder and denoiser respectively:

ℒanchorTE\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}^\{\\mathrm\{TE\}\}=∑c∈𝒞Sλc​‖𝐞c\(0\)−𝐞c‖2,\\displaystyle=\\sum\_\{c\\in\\mathcal\{C\}\_\{S\}\}\\lambda\_\{c\}\\,\\bigl\\\|\\mathbf\{e\}^\{\(0\)\}\_\{c\}\-\\mathbf\{e\}\_\{c\}\\bigr\\\|^\{2\},\(13\)ℒanchor𝒰\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}^\{\\mathcal\{U\}\}=∑c∈𝒞Sλc​‖𝐀𝒰,\(0\)​\[c,:\]−𝐀𝒰​\[c,:\]‖2,\\displaystyle=\\sum\_\{c\\in\\mathcal\{C\}\_\{S\}\}\\lambda\_\{c\}\\,\\bigl\\\|\\mathbf\{A\}^\{\\mathcal\{U\},\(0\)\}\[c,\\,:\]\-\\mathbf\{A\}^\{\\mathcal\{U\}\}\[c,\\,:\]\\bigr\\\|^\{2\},\(14\)where𝐞c\(0\)\\mathbf\{e\}^\{\(0\)\}\_\{c\}and𝐀𝒰,\(0\)​\[c,:\]\\mathbf\{A\}^\{\\mathcal\{U\},\(0\)\}\[c,\\,:\]are the embedding and CG row ofccunder the frozen base model\.

Preservation loss\.To prevent catastrophic forgetting of concepts outside𝒞\\mathcal\{C\}, we augment with the standard LDM noise\-prediction loss on reference pairs𝒟pres\\mathcal\{D\}\_\{\\mathrm\{pres\}\}:

ℒpres=𝔼\(x0,c~\),ϵ,t​\[‖ϵ−ϵθ​\(α¯t​z0\+1−α¯t​ϵ,fϕ​\(c~\),t\)‖2\],\\mathcal\{L\}\_\{\\mathrm\{pres\}\}=\\mathbb\{E\}\_\{\(x\_\{0\},\\tilde\{c\}\),\\epsilon,t\}\\\!\\left\[\\left\\\|\\epsilon\-\\epsilon\_\{\\theta\}\\\!\\left\(\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,z\_\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\epsilon,\\;f\_\{\\phi\}\(\\tilde\{c\}\),\\;t\\right\)\\right\\\|^\{2\}\\right\],\(15\)wherez0=ℰ​\(x0\)z\_\{0\}=\\mathcal\{E\}\(x\_\{0\}\)is the VAE\-encoded latent\.

Total objective\.

ℒ=δ⋅ℒrank\+γ⋅ℒpres\+ℒanchor,\\mathcal\{L\}=\\delta\\cdot\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\+\\gamma\\cdot\\mathcal\{L\}\_\{\\mathrm\{pres\}\}\+\\mathcal\{L\}\_\{\\mathrm\{anchor\}\},\(16\)whereδ,γ≥0\\delta,\\gamma\\geq 0balance the relative contributions\.

## 6Experiments and Results

We now empirically evaluate our method, assessing both its effectiveness at mitigating bias and its impact on generation quality\.

Experimental Setup

Models and datasets\.All experiments use Stable Diffusion v1\.5\(Rombachet al\.,[2022](https://arxiv.org/html/2607.03397#bib.bib35)\)as the base generative model\. For the debiasing evaluation we follow the protocol of DiffLens\(Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\): we generate500500images per occupation prompt \(“face of a/an\{occupation\}”\) across four occupations, namely,nurse,doctor,fire fighter, andreceptionist, and evaluate three demographic axes: gender \(Male/Female\), race \(White, Black, Asian, Indian\), and age \(Young,Adult,Old\)\. Demographic labels are obtained from the FairFace\(Kärkkäinen and Joo,[2021](https://arxiv.org/html/2607.03397#bib.bib52)\)4\-race ResNet\-34 classifier\.

Baselines and metrics\.We compare against five state\-of\-the\-art debiasing baselines:Asyrp\(Kwonet al\.,[2023b](https://arxiv.org/html/2607.03397#bib.bib36)\),H\-Distribution\(Pariharet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib16)\),Latent Directions\(Liet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib17)\),Finetuning\(Shenet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib23)\), andDiffLens\(Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\)\. We compute four metrics for comparing the fairness, and generation quality ofCO\-ALIGNagainst the baselines\. We compute Fairness Discrepancy \(FD\)\(Pariharet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib16)\)for evaluating the fairness of debiasing, FID\(Heuselet al\.,[2017](https://arxiv.org/html/2607.03397#bib.bib55)\)for evauating the photorealism, and CLIP\-I, and CLIP\-T scores to evaluate the fidelity of generation\. We discuss these metrics in detail in Appendix[D](https://arxiv.org/html/2607.03397#A4)

Evaluation Results

Fairness\.CO\-ALIGNachieves the lowest FD for gender \(FD=0\.032\\text\{FD\}=0\.032,30\.4%30\.4\\%improvement over the next best\) and race \(FD=0\.043\\text\{FD\}=0\.043,75\.4%75\.4\\%improvement over the next best\), and a competitive FD for age \(FD=0\.054\\text\{FD\}=0\.054\), on par with DiffLens\.

Generation quality\.On image qualityCO\-ALIGNachievesΔ​FID=11\.4\\Delta\\mathrm\{FID\}=11\.4below the base model average, outperforming all baselines on FID\. CLIP\-I scores of0\.95060\.9506\(gender\),0\.95250\.9525\(age\),0\.91220\.9122\(race\) confirm that the aligned model’s outputs remain close to those of the base model, representing a2\.8%2\.8\\%improvement in image fidelity over the strongest baseline\. CLIP\-T scores are stable across all methods, indicating that prompt adherence is preserved throughout alignment, and is the best across all baselines\.

Incoherence\.A distinctive failure mode of single\-component debiasing issemantic incoherence: generations that fail to depict the target concept, arising from the distribution shift between the edited and unedited components, measured as the fraction of generated images classified by LLaVA as off\-target\. Figure[2](https://arxiv.org/html/2607.03397#S3.F2)\(b\) shows that both text\-encoder\-only \(Debias\-VL, Finetuning\) and denoiser\-only \(UCE, Latent Directions\) methods trade fairness for incoherence, whileCO\-ALIGNis the only method in the ideal bottom\-left region, reducing incoherence by88%over the base model while achieving the best fairness\. Figure[4](https://arxiv.org/html/2607.03397#S6.F4)further illustrates this: text\-encoder alignment alone produces incoherent outputs, which denoiser alignment resolves by reconfiguring the denoiser’s concept graph to match the updated embedding geometry\.

Table 2:Evaluation of bias mitigation in text\-to\-image diffusion model Stable DiffusionRombachet al\.\([2022](https://arxiv.org/html/2607.03397#bib.bib35)\), based on average performance across four occupations\. We highlight the best results inboldand the second\-best withunderlinedtext \(excluding the “original”\)\.MethodGender \(2\)Age \(3\)Race \(4\)FD↓\\downarrowFID↓\\downarrowCLIP\-I↑\\uparrowCLIP\-T↑\\uparrowFD↓\\downarrowFID↓\\downarrowCLIP\-I↑\\uparrowCLIP\-T↑\\uparrowFD↓\\downarrowFID↓\\downarrowCLIP\-I↑\\uparrowCLIP\-T↑\\uparrowOriginal0\.564120\.06–0\.61550\.752120\.06–0\.61550\.558120\.06–0\.6155AsyrpKwonet al\.\([2023b](https://arxiv.org/html/2607.03397#bib.bib36)\)0\.408166\.110\.82530\.60050\.682200\.900\.85270\.61220\.524153\.060\.88040\.6086H\-DistributionPariharet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib16)\)0\.222151\.680\.84750\.60870\.506147\.710\.83450\.60980\.544126\.900\.82550\.6100Latent DirectionLiet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib17)\)0\.305129\.370\.80580\.60910\.052113\.810\.81510\.60670\.175128\.300\.82110\.6132FinetuningShenet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib23)\)0\.050161\.470\.87790\.60950\.746161\.470\.87790\.60950\.198161\.470\.87790\.6095DiffLensShiet al\.\([2025](https://arxiv.org/html/2607.03397#bib.bib19)\)0\.046112\.830\.85010\.60900\.04999\.170\.87780\.60570\.401119\.860\.90960\.6149UCEGandikotaet al\.\([2024](https://arxiv.org/html/2607.03397#bib.bib14)\)0\.4117132\.600\.78740\.61550\.5792122\.860\.91690\.61360\.2073110\.940\.93380\.6144CO\-ALIGN\(Ours\)0\.03298\.880\.95060\.61530\.05497\.420\.95250\.61920\.043101\.450\.91220\.6219

![Refer to caption](https://arxiv.org/html/2607.03397v1/x4.png)Figure 4:The figure shows how alignment of just the text encoder leads to generation of incoherent output due to mismatch in representations between the text encoder and the denoising unit\. The subsequent alignment of the denoiser reconfigures the representation geometry between the text encoder and the denoiser, thus regaining coherence\.
## 7Beyond Bias Mitigation: a Case Study on Unlearning

Post\-hoc concept unlearning techniques erase a target conceptcTc\_\{T\}by editing model parameters\(Gandikotaet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib56),[2024](https://arxiv.org/html/2607.03397#bib.bib14)\)or steering its latent representation\(Yoonet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib57)\)\. However, only concepts within alocality radiusofcTc\_\{T\}in concept space are erased; adversarial bypass concepts outside this neighbourhood𝒩​\(cT\)\\mathcal\{N\}\(c\_\{T\}\)remain unaffected, allowing attackers to recover unlearned content via semantically adjacent prompts\.

Table 3:Percentage of images flagged as nudity by VLM classifier under adversarial prompt categories \(lower is better\)\.Bolddenotes best result per column within each group\.CO\-ALIGNpre\-aligns adversarial concepts closer to nudity in the model’s concept space, substantially boosting the effectiveness of post\-hoc unlearning\.Base ModelUnlearningTechniquePornStarNymph\.Cream\.Shirt\.Att\.Fem\.AvgSD v1\.5None14\.062\.0100\.040\.026\.048\.4\+\+UCE2\.040\.088\.010\.08\.029\.6\+\+ESD2\.014\.012\.018\.04\.010\.0\+\+SAFREE0\.00\.094\.00\.00\.018\.8SD v1\.5\+\+CO\-ALIGN\(Ours\)None16\.0100\.0100\.060\.026\.060\.4\[2pt/2pt\]\+\+UCE0\.018\.032\.00\.00\.010\.0\+\+ESD0\.00\.02\.02\.00\.00\.8\+\+SAFREE0\.04\.00\.02\.00\.01\.2We exploitCO\-ALIGN’s concept topology editing to address this vulnerability\. Before unlearning, we useCO\-ALIGNto pull a set of known adversarial concepts𝒞adv\\mathcal\{C\}\_\{\\mathrm\{adv\}\}\(e\.g\., forcT=Nudityc\_\{T\}=\\textit\{Nudity\}:𝒞adv=\{Nymphettes,Creampie\}\\mathcal\{C\}\_\{\\mathrm\{adv\}\}=\\\{\\textit\{Nymphettes\},\\,\\textit\{Creampie\}\\\}\) closer tocTc\_\{T\}in both the text encoder and denoiser concept graphs, bringing them within𝒩​\(cT\)\\mathcal\{N\}\(c\_\{T\}\)before any unlearning technique is applied\.

Neighbourhood Pulling Effect

A key emergent property of concept graph alignment issemantic neighbourhood propagation: aligning a supervised conceptc∗∈𝒞advc^\{\*\}\\in\\mathcal\{C\}\_\{\\mathrm\{adv\}\}towardcTc\_\{T\}causes its unsupervised semantic neighbors to shift towardcTc\_\{T\}as well, without any direct supervision signal\. Formally, let𝒩ϵ​\(c∗\)\\mathcal\{N\}\_\{\\epsilon\}\(c^\{\*\}\)denote the set of concepts within concept distanceϵ\\epsilonofc∗c^\{\*\}in the model’s concept graph before alignment\. After aligningc∗c^\{\*\}towardcTc\_\{T\}viaℒrank\\mathcal\{L\}\_\{\\mathrm\{rank\}\}, we observe that for allcn∈𝒩ϵ​\(c∗\)c\_\{n\}\\in\\mathcal\{N\}\_\{\\epsilon\}\(c^\{\*\}\):

Δ\|cn,cT\|C​G=\|cn,cT\|C​Gafter−\|cn,cT\|C​Gbefore\>0,\\Delta\|c\_\{n\},\\,c\_\{T\}\|\_\{CG\}=\|c\_\{n\},c\_\{T\}\|\_\{CG\_\{\{\\mathrm\{after\}\}\}\}\-\|c\_\{n\},c\_\{T\}\|\_\{CG\_\{\{\\mathrm\{before\}\}\}\}\>0,\(17\)even thoughcnc\_\{n\}received no direct alignment supervision\. This effect is analogous to label propagation in graph\-based semi\-supervised learning\(Zhouet al\.,[2003](https://arxiv.org/html/2607.03397#bib.bib43); Zhuet al\.,[2003](https://arxiv.org/html/2607.03397#bib.bib44); Iscenet al\.,[2019](https://arxiv.org/html/2607.03397#bib.bib45)\): the graph smoothness assumption propagates alignment from supervised nodes to adjacent unsupervised ones\.

Figure[5](https://arxiv.org/html/2607.03397#S7.F5)visualizes this effect for both the text encoder and the UNet\. Each line traces a concept’s similarity to Nudity before \(z=0=0\) and after \(z=1=1\)CO\-ALIGNalignment\. Red lines denote the two directly supervised concepts \(Nymphettes, Creampie\); blue lines denote unsupervised concepts \(Erotic, Attractive Female, Jonny Sins, Shirtless\)\. Despite receiving no direct supervision, all unsupervised concepts exhibit a measurable upward shift in similarity to Nudity \(\+0\.15\+0\.15to\+0\.22\+0\.22in the UNet graph\), confirming that alignment of𝒞adv\\mathcal\{C\}\_\{\\mathrm\{adv\}\}broadcasts into the surrounding concept neighborhood\.

![Refer to caption](https://arxiv.org/html/2607.03397v1/x5.png)Figure 5:The figure shows the Semantic Neighborhood Pulling Effect on Concept Graphs as a result ofCO\-ALIGNedit for both the Text encoder and the Unet\. The Y\-axis measures distance to Nudity; the X\-axis reflects inter\-concept neighbourhood structure via 1\-D MDS \(concepts closer in embedding space appear closer on X\)\.CO\-ALIGNfor Robust Unlearning

For the unlearning evaluation, we generate500500images per adversarial concept and classify nudity presence with LLaVA\(Liuet al\.,[2023a](https://arxiv.org/html/2607.03397#bib.bib58)\), following the prominent adversarial concepts from I2P dataset\(Schramowskiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib59)\)\. We evaluate three unlearning methodsUCE\(Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\),ESD\(Gandikotaet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib56)\), andSAFREE\(Yoonet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib57)\)applied both to the base model and to aCO\-ALIGN\-aligned model\.

Procedure\.The CO\-ALIGN pre\-alignment brings𝒞adv\\mathcal\{C\}\_\{\\mathrm\{adv\}\}and, via semantic neighborhood propagation, a broader set of unsupervised bypass concepts within the locality radius𝒩​\(cT\)\\mathcal\{N\}\(c\_\{T\}\)before unlearning is applied\. Any subsequent unlearning technique that targetscTc\_\{T\}therefore also covers the pre\-aligned bypass concepts, and their neighboring concepts, without requiring knowledge of those concepts at unlearning time\.

Robustness of unlearning withCO\-ALIGN\.Table[3](https://arxiv.org/html/2607.03397#S7.T3)reports the percentage of images flagged as nudity by the VLM classifier under five adversarial prompt categories\. Across all three unlearning techniques, pre\-aligning withCO\-ALIGNsubstantially improves adversarial robustness\.CO\-ALIGN\+ ESD reduces the average nudity rate from10\.0%10\.0\\%to0\.8%\\mathbf\{0\.8\\%\};CO\-ALIGN\+ SAFREE from18\.8%18\.8\\%to1\.2%\\mathbf\{1\.2\\%\}; andCO\-ALIGN\+ UCE from29\.6%29\.6\\%to10\.0%\\mathbf\{10\.0\\%\}\. Notably, the Creampie prompt which achieves88%88\\%\-100%100\\%nudity generation against all baselines, drops to0%0\\%\-32%32\\%afterCO\-ALIGNpre\-alignment, demonstrating that neighborhood propagation successfully relocates this adversarial bypass concept into the unlearning target’s erasure zone\.

## 8Conclusion

We presentedCO\-ALIGN, a bias mitigation framework that jointly aligns concept graphs extracted from the text encoder and denoiser toward a target topology\. Our central hypothesis, that effective debiasing requires geometric harmony between both components, is confirmed empirically: single\-component methods lie off the Pareto frontier of fairness and coherence, whileCO\-ALIGNachieves a30%30\\%fairness improvement,Δ​FID=11\.4\\Delta\\text\{FID\}=11\.4,2\.8%2\.8\\%higher image fidelity, and an88%88\\%reduction in semantically incoherent outputs, all without added inference cost\. Beyond debiasing, the emergent neighborhood propagation effect improves adversarial robustness of post\-hoc unlearning by up to12\.5×12\.5\\timeswithout modifying the unlearning procedure\. Limitations include the compute cost of denoiser graph extraction and the need to re\-derive concept graph construction for transformer\-based architectures such as DiT\(Peebles and Xie,[2023](https://arxiv.org/html/2607.03397#bib.bib60)\)and SDXL\(Podellet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib61)\), which we leave for future work\.

## References

- S\. Basu, K\. Rezaei, P\. Kattakinda, V\. I\. Morariu, N\. Zhao, R\. A\. Rossi, V\. Manjunatha, and S\. Feizi \(2024\)On mechanistic knowledge localization in text\-to\-image generative models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§4](https://arxiv.org/html/2607.03397#S4.p1.3)\.
- Manifold regularization: a geometric framework for learning from labeled and unlabeled examples\.InJournal of Machine Learning Research,Vol\.7,pp\. 2399–2434\.Cited by:[§1](https://arxiv.org/html/2607.03397#S1.p4.1)\.
- F\. Bianchi, P\. Kalluri, E\. Durmus, F\. Ladhak, M\. Cheng, D\. Nozza, T\. Hashimoto, D\. Jurafsky, J\. Zou, and A\. Caliskan \(2023\)Easily accessible text\-to\-image generation amplifies demographic stereotypes at large scale\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency,FAccT ’23,New York, NY, USA,pp\. 1–12\.External Links:[Document](https://dx.doi.org/10.1145/3593013.3594095)Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- M\. Brack, F\. Friedrich, D\. Hintersdorf, L\. Struppek, P\. Schramowski, and K\. Kersting \(2023\)SEGA: instructing text\-to\-image models using semantic guidance\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1)\.
- A\. T\. Bui, T\. Vu, L\. T\. Vuong, T\. Le, P\. Montague, T\. Abraham, J\. Kim, and D\. Phung \(2025\)Fantastic targets for concept erasure in diffusion models and where to find them\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tZdqL5FH7w)Cited by:[§1](https://arxiv.org/html/2607.03397#S1.p4.1)\.
- R\. Chakraborty, X\. Che, C\. Faklaris, X\. Niu, D\. Xu, and S\. Yuan \(2025\)BiasMap: leveraging cross\-attentions to discover and mitigate hidden social biases in text\-to\-image generation\.arXiv preprint arXiv:2509\.13496\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.12.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- J\. Cho, A\. Zala, and M\. Bansal \(2023\)DALL\-Eval: probing the reasoning skills and social biases of text\-to\-image generation models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3043–3054\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- C\. Chuang, V\. Jampani, Y\. Li, A\. Torralba, and S\. Jegelka \(2023\)Debiasing vision\-language models via biased prompts\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[9th item](https://arxiv.org/html/2607.03397#Ax1.I1.i9.p1.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.3.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1),[§3](https://arxiv.org/html/2607.03397#S3.p2.1),[§3](https://arxiv.org/html/2607.03397#S3.p4.1)\.
- C\. Fan, J\. Liu, Y\. Zhang, E\. Wong, D\. Wei, and S\. Liu \(2024\)SalUn: empowering machine unlearning via gradient\-based weight saliency in both image classification and generation\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gn0mIhQGNM)Cited by:[§4](https://arxiv.org/html/2607.03397#S4.p1.3)\.
- F\. Friedrich, M\. Brack, L\. Struppek, D\. Hintersdorf, P\. Schramowski, S\. Luccioni, and K\. Kersting \(2023\)Fair diffusion: instructing text\-to\-image generation models on fairness\.arXiv preprint arXiv:2302\.10893\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.2.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1)\.
- R\. Gandikota, J\. Materzyńska, J\. Fiotto\-Kaufman, and D\. Bau \(2023\)Erasing concepts from diffusion models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 2400–2410\.Cited by:[7th item](https://arxiv.org/html/2607.03397#Ax1.I1.i7.p1.1),[§7](https://arxiv.org/html/2607.03397#S7.p1.3),[§7](https://arxiv.org/html/2607.03397#S7.p7.1)\.
- R\. Gandikota, H\. Orgad, Y\. Belinkov, J\. Materzyńska, and D\. Bau \(2024\)Unified concept editing in diffusion models\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 5090–5100\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Appendix C](https://arxiv.org/html/2607.03397#A3.p2.11),[7th item](https://arxiv.org/html/2607.03397#Ax1.I1.i7.p1.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.16.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1),[§3](https://arxiv.org/html/2607.03397#S3.p4.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.20.1),[§7](https://arxiv.org/html/2607.03397#S7.p1.3),[§7](https://arxiv.org/html/2607.03397#S7.p7.1)\.
- M\. Heusel, H\. Ramsauer, T\. Unterthiner, B\. Nessler, and S\. Hochreiter \(2017\)GANs trained by a two time\-scale update rule converge to a local Nash equilibrium\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[2nd item](https://arxiv.org/html/2607.03397#A4.I1.i2.p1.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- A\. Iscen, G\. Tolias, Y\. Avrithis, and O\. Chum \(2019\)Label propagation for deep semi\-supervised learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 5070–5079\.Cited by:[§1](https://arxiv.org/html/2607.03397#S1.p4.1),[§7](https://arxiv.org/html/2607.03397#S7.p4.11)\.
- M\. Kang, V\. B\. Kumar, S\. Roy, A\. Kumar, S\. Khosla, B\. M\. Narayanaswamy, and R\. Gangadharaiah \(2025\)FairGen: controlling sensitive attributes for fair generations in diffusion models via adaptive latent guidance\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 25336–25350\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.4.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1)\.
- K\. Kärkkäinen and J\. Joo \(2021\)FairFace: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation\.In2021 IEEE Winter Conference on Applications of Computer Vision \(WACV\),Vol\.,pp\. 1547–1557\.External Links:[Document](https://dx.doi.org/10.1109/WACV48630.2021.00159)Cited by:[4th item](https://arxiv.org/html/2607.03397#Ax1.I1.i4.p1.1),[§3](https://arxiv.org/html/2607.03397#S3.p1.1),[§6](https://arxiv.org/html/2607.03397#S6.p3.1)\.
- E\. Kim, S\. Kim, M\. Park, R\. Entezari, and S\. Yoon \(2025\)Rethinking training for de\-biasing text\-to\-image generation: unlocking the potential of stable diffusion\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.5.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1)\.
- M\. Kwon, J\. Jeong, and Y\. Uh \(2023a\)Diffusion models already have a semantic latent space\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.13.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- M\. Kwon, J\. Jeong, and Y\. Uh \(2023b\)Diffusion models already have a semantic latent space\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pd1P2eUBVfq)Cited by:[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.15.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- H\. Li, C\. Shen, P\. Torr, V\. Tresp, and J\. Gu \(2024\)Self\-discovering interpretable diffusion latent directions for responsible text\-to\-image generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12006–12016\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.8.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1),[§3](https://arxiv.org/html/2607.03397#S3.p4.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.17.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- J\. Li, L\. Hu, J\. Zhang, T\. Zheng, H\. Zhang, and D\. Wang \(2025\)Fair text\-to\-image diffusion via fair mapping\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 26256–26264\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i25.34823)Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.14.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1)\.
- T\. Linet al\.\(2014\)Microsoft coco: common objects in context\.InECCV,Cited by:[11st item](https://arxiv.org/html/2607.03397#Ax1.I1.i11.p1.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[Appendix F](https://arxiv.org/html/2607.03397#A6.p1.2),[5th item](https://arxiv.org/html/2607.03397#Ax1.I1.i5.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023a\)Visual instruction tuning\.External Links:2304\.08485,[Link](https://arxiv.org/abs/2304.08485)Cited by:[§7](https://arxiv.org/html/2607.03397#S7.p7.1)\.
- Z\. Liu, R\. Feng, K\. Zhu, Y\. Zhang, K\. Zheng, Y\. Liu, D\. Zhao, J\. Zhou, and Y\. Cao \(2023b\)Cones: concept neurons in diffusion models for customized generation\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§4](https://arxiv.org/html/2607.03397#S4.p1.3)\.
- A\. S\. Luccioni, C\. Akiki, M\. Mitchell, and Y\. Jernite \(2023\)Stable bias: analyzing societal representations in diffusion models\.arXiv preprint arXiv:2303\.11408\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- Mansi, A\. Kori, F\. Toni, and S\. Demetriou \(2026\)Selective fine\-tuning for targeted and robust concept unlearning\.External Links:2602\.07919,[Link](https://arxiv.org/abs/2602.07919)Cited by:[§4](https://arxiv.org/html/2607.03397#S4.p1.3)\.
- R\. Naik and B\. Nushi \(2023\)Social biases through the text\-to\-image generation lens\.arXiv preprint arXiv:2304\.06034\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- H\. Orgad, B\. Kawar, and Y\. Belinkov \(2023\)Editing implicit assumptions in text\-to\-image diffusion models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 7053–7061\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Appendix C](https://arxiv.org/html/2607.03397#A3.p2.11),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.15.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1),[§3](https://arxiv.org/html/2607.03397#S3.p2.1)\.
- R\. Parihar, A\. Bhat, A\. Basu, S\. Mallick, J\. N\. Kundu, and R\. V\. Babu \(2024\)Balancing act: distribution\-guided debiasing in diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 6668–6678\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[1st item](https://arxiv.org/html/2607.03397#A4.I1.i1.p1.4),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.7.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.16.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- J\. Park, J\. Lee, C\. Chung, J\. Lee, J\. Choo, and J\. Gu \(2025\)Fair generation without unfair distortions: debiasing text\-to\-image generation with entanglement\-free attention\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.6.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- W\. Peebles and S\. Xie \(2023\)Scalable diffusion models with transformers\.In2023 IEEE/CVF International Conference on Computer Vision \(ICCV\),Vol\.,pp\. 4172–4182\.External Links:[Document](https://dx.doi.org/10.1109/ICCV51070.2023.00387)Cited by:[§8](https://arxiv.org/html/2607.03397#S8.p1.5)\.
- M\. V\. Perera and V\. M\. Patel \(2023a\)Analyzing bias in diffusion\-based face generation models\.In2023 IEEE International Joint Conference on Biometrics \(IJCB\),pp\. 1–10\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1),[§3](https://arxiv.org/html/2607.03397#S3.p1.1)\.
- M\. V\. Perera and V\. M\. Patel \(2023b\)Analyzing bias in diffusion\-based face generation models\.External Links:2305\.06402,[Link](https://arxiv.org/abs/2305.06402)Cited by:[§3](https://arxiv.org/html/2607.03397#S3.p1.1)\.
- D\. Podell, Z\. English, K\. Lacey, A\. Blattmann, T\. Dockhorn, J\. Müller, J\. Penna, and R\. Rombach \(2024\)SDXL: improving latent diffusion models for high\-resolution image synthesis\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=di52zR8xgf)Cited by:[§8](https://arxiv.org/html/2607.03397#S8.p1.5)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\-24 July 2021, Virtual Event,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.External Links:[Link](http://proceedings.mlr.press/v139/radford21a.html)Cited by:[Appendix C](https://arxiv.org/html/2607.03397#A3.p1.8),[3rd item](https://arxiv.org/html/2607.03397#A4.I1.i3.p1.1),[3rd item](https://arxiv.org/html/2607.03397#Ax1.I1.i3.p1.1)\.
- R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer \(2022\)High\-Resolution Image Synthesis with Latent Diffusion Models\.In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\. 10674–10685\.External Links:ISSN,[Document](https://dx.doi.org/10.1109/CVPR52688.2022.01042),[Link](https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.01042)Cited by:[Appendix C](https://arxiv.org/html/2607.03397#A3.p1.3),[1st item](https://arxiv.org/html/2607.03397#Ax1.I1.i1.p1.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2),[§6](https://arxiv.org/html/2607.03397#S6.p3.1)\.
- N\. Roos, E\. Iakovleva, A\. Gjergji, V\. P\. Pastore, and E\. Tartaglione \(2026\)How I met your bias: investigating bias amplification in diffusion models\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- P\. Schramowski, M\. Brack, B\. Deiseroth, and K\. Kersting \(2023\)Safe latent diffusion: mitigating inappropriate degeneration in diffusion models\.In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 22522–22531\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52729.2023.02157)Cited by:[§7](https://arxiv.org/html/2607.03397#S7.p7.1)\.
- C\. Schuhmann, R\. Beaumont, R\. Vencu, C\. W\. Gordon, R\. Wightman, M\. Cherti, T\. Coombes, A\. Katta, C\. Mullis, M\. Wortsman, P\. Schramowski, S\. R\. Kundurthy, K\. Crowson, L\. Schmidt, R\. Kaczmarczyk, and J\. Jitsev \(2022a\)LAION\-5b: an open large\-scale dataset for training next generation image\-text models\.InThirty\-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=M3Y74vmsMcY)Cited by:[§3](https://arxiv.org/html/2607.03397#S3.p1.1)\.
- C\. Schuhmann, R\. Beaumont, R\. Vencu, C\. Gordon, R\. Wightman, M\. Cherti, T\. Coombes, A\. Katta, C\. Mullis, M\. Wortsman, P\. Schramowski, S\. Kundurthy, K\. Crowson, L\. Schmidt, R\. Kaczmarczyk, and J\. Jitsev \(2022b\)LAION\-5B: an open large\-scale dataset for training next generation image\-text models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 25278–25294\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1)\.
- P\. Seshadri, S\. Singh, and Y\. Elazar \(2024\)The bias amplification paradox in text\-to\-image generation\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 6367–6384\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.353)Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p1.1),[§2](https://arxiv.org/html/2607.03397#S2.p1.1),[§3](https://arxiv.org/html/2607.03397#S3.p1.1)\.
- X\. Shen, C\. Du, T\. Pang, M\. Lin, Y\. Wong, and M\. Kankanhalli \(2024\)Finetuning text\-to\-image diffusion models for fairness\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p3.1),[10th item](https://arxiv.org/html/2607.03397#Ax1.I1.i10.p1.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.17.1),[§2](https://arxiv.org/html/2607.03397#S2.p3.1),[§3](https://arxiv.org/html/2607.03397#S3.p4.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.18.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- Y\. Shi, C\. Li, Y\. Wang, Y\. Zhao, A\. Pang, S\. Yang, J\. Yu, and K\. Ren \(2025\)Dissecting and mitigating diffusion bias via mechanistic interpretability\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 8192–8202\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[§1](https://arxiv.org/html/2607.03397#S1.p1.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.11.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1),[§3](https://arxiv.org/html/2607.03397#S3.p3.1),[Table 2](https://arxiv.org/html/2607.03397#S6.T2.12.12.19.1),[§6](https://arxiv.org/html/2607.03397#S6.p3.1),[§6](https://arxiv.org/html/2607.03397#S6.p4.1)\.
- K\. S\. Vardhana, S\. Lolla, and S\. Biswas \(2026\)Fully unsupervised self\-debiasing of text\-to\-image diffusion models\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.9.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- P\. von Platenet al\.\(2022\)Diffusers: state\-of\-the\-art diffusion models\.External Links:[Link](https://github.com/huggingface/diffusers)Cited by:[2nd item](https://arxiv.org/html/2607.03397#Ax1.I1.i2.p1.1)\.
- Y\. Wu, Y\. Nakashima, and N\. Garcia \(2025\)Revealing gender bias from prompt to image in stable diffusion\.Journal of Imaging11\(2\),pp\. 35\.External Links:[Document](https://dx.doi.org/10.3390/jimaging11020035)Cited by:[§3](https://arxiv.org/html/2607.03397#S3.p3.1)\.
- A\. Yasser, K\. Phunjanna, M\. Escudero Viñolo, C\. Barata, and J\. Benois\-Pineau \(2026\)Locating demographic bias at the attention\-head level in CLIP’s vision encoder\.arXiv preprint arXiv:2603\.11793\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- J\. Yoon, S\. Yu, V\. Patil, H\. Yao, and M\. Bansal \(2025\)SAFREE: training\-free and adaptive guard for safe text\-to\-image and video generation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hgTFotBRKl)Cited by:[8th item](https://arxiv.org/html/2607.03397#Ax1.I1.i8.p1.1),[§7](https://arxiv.org/html/2607.03397#S7.p1.3),[§7](https://arxiv.org/html/2607.03397#S7.p7.1)\.
- E\. Z\. Zeng, Y\. Chen, and A\. Wong \(2026\)SCALEX: scalable concept and latent exploration for diffusion models\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.10.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- D\. Zhou, O\. Bousquet, T\. N\. Lal, J\. Weston, and B\. Schölkopf \(2003\)Learning with local and global consistency\.InAdvances in Neural Information Processing Systems,Vol\.16\.Cited by:[§1](https://arxiv.org/html/2607.03397#S1.p4.1),[§7](https://arxiv.org/html/2607.03397#S7.p4.11)\.
- J\. Zhou, J\. Gao, X\. Zhao, X\. Yao, and X\. Wei \(2024\)Association of objects may engender stereotypes: mitigating association\-engendered stereotypes in text\-to\-image generation\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2607.03397#A1.p4.1),[Table 1](https://arxiv.org/html/2607.03397#S2.T1.3.1.18.1),[§2](https://arxiv.org/html/2607.03397#S2.p4.1)\.
- X\. Zhu, Z\. Ghahramani, and J\. Lafferty \(2003\)Semi\-supervised learning using gaussian fields and harmonic functions\.InProceedings of the Twentieth International Conference on Machine Learning,pp\. 912–919\.Cited by:[§1](https://arxiv.org/html/2607.03397#S1.p4.1),[§7](https://arxiv.org/html/2607.03397#S7.p4.11)\.

## Appendix AInherent Biases in T2I Diffusion Models

![Refer to caption](https://arxiv.org/html/2607.03397v1/x6.png)Figure 6:Demographic bias on randomly sampled 10k images from LAION 2B dataset\.Dataset biases and their implications\.Alike other generative models, biases in T2I diffusion models often originate from the large\-scale web\-scraped corpora used for training\. LAION\-5B\[Schuhmannet al\.,[2022b](https://arxiv.org/html/2607.03397#bib.bib1)\], used for training Stable Diffusion, mirrors societal inequalities at internet scale across profession\[Bianchiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib2)\], race\[Choet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib3), Luccioniet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib33)\]and culture\[Naik and Nushi,[2023](https://arxiv.org/html/2607.03397#bib.bib4)\]\. These biases have shown to be*amplified*beyond the degree present in training data both during model training\[Seshadriet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib5), Perera and Patel,[2023a](https://arxiv.org/html/2607.03397#bib.bib6)\]and inference\[Rooset al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib8)\]\. Large scale deployment of these models in both human operated and automated systems have shown detrimental societal consequences of reinforcement of stereotypes\[Bianchiet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib2)\], across gender, age, race, and geography simultaneously\[Naik and Nushi,[2023](https://arxiv.org/html/2607.03397#bib.bib4)\]\.

Bias mitigation techniques\.Existing debiasing techniques can be broadly classified in two groups, depending on where in the generation pipeline they intervene, as summarised in Table[1](https://arxiv.org/html/2607.03397#S2.T1)\.

Text encoder debiasingmethods intervene solely at the text encoder\. Shen et al\.\[Shenet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib23)\]fine\-tune the text encoder directly using a distributional alignment loss on generated images\. Fair Mapping\[Liet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib40)\]learns a lightweight linear remap over conditioning embeddings to project them into a debiased subspace\. Chuang et al\.\[Chuanget al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib11)\]achieve a similar effect training\-free, via a calibrated projection matrix that removes biased directions from text embeddings at inference time\. Fair Diffusion\[Friedrichet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib9)\]via SEGA\[Bracket al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib10)\], FairGen\[Kanget al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib39)\], and Kim et al\.\[Kimet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib12)\]take a softer approach, steering the conditioning signal or noise initialisation at inference without any weight modification\. While efficient, all text\-encoder\-only methods leave the denoiser’s internal concept organization and the visual biases encoded\.

Denoiser debiasingmethods instead target just the denoiser’s internal representations\. TIME\[Orgadet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib13)\]edits the key and value projection matrices in the UNet’s cross\-attention layers in closed form, rerouting underspecified prompts toward target attribute embeddings\. UCE\[Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\]extends this to performing multiple edits while only targeting the value projection matrices\. Asyrp\[Kwonet al\.,[2023a](https://arxiv.org/html/2607.03397#bib.bib15)\]establishes the UNet bottleneck \(h\-space\) as a semantically linear latent space amenable to controllable editing\. Building on this, Balancing Act\[Pariharet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib16)\]train a lightweight Attribute Distribution Predictor on h\-space features to guide debiased generation without weight modification, while Li et al\.\[Liet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib17)\]discover interpretable fairness\-sensitive directions in h\-space without external classifiers\. Vardhana et al\.\[Vardhanaet al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib18)\]perform fully unsupervised debiasing by clustering image encoder embeddings and steering denoising toward a uniform attribute distribution\. SCALEX\[Zenget al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib22)\]maps the conceptual structure of diffusion models in h\-space via prompt\-aligned latent directions derived from Latent Consistency Models\. DIFFLENS\[Shiet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib19)\]takes a more mechanistic approach, decomposing h\-space activations via sparse autoencoders to identify and suppress specific neuron\-level dimensions responsible for bias\. BiasMap\[Chakrabortyet al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib20)\]and EFA\[Parket al\.,[2025](https://arxiv.org/html/2607.03397#bib.bib37)\]further reveal that many denoiser\-side corrections reduce output distributional gaps without disentangling the underlying concept coupling in cross\-attention, leaving residual structural bias intact\. Yasser et al\.\[Yasseret al\.,[2026](https://arxiv.org/html/2607.03397#bib.bib21)\]extend this mechanistic analysis to the CLIP vision encoder, identifying specific attention heads whose ablation reduces gender bias more surgically than age bias\. MAS\[Zhouet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib38)\]addresses the related problem of association\-engendered stereotypes arising from the co\-generation of multiple concepts\. While these methods edit the denoiser’s internal representations, they are applied independently of the text encoder, leaving the biased embedding geometry of underspecified prompts uncorrected\.

These two lines of work are thus complementary but incomplete: text\-encoder methods leave visual biases in the denoiser intact, while denoiser\-side methods inherit a biased conditioning signal\. This motivates a global approach that jointly debiases both components\.

## Appendix BAlgorithms

Algorithm 1Text Encoder Concept Graph Extraction1:Concept set

𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}, text encoder

fϕf\_\{\\phi\}
2:Adjacency matrix

𝐀TE∈ℝn×n\\mathbf\{A\}^\{\\mathrm\{TE\}\}\\in\\mathbb\{R\}^\{n\\times n\}
3:for

i=1i=1to

nndo

4:Tokenise

cic\_\{i\}and compute hidden states via

fϕf\_\{\\phi\}
5:

𝐞i←MeanPool​\(LastHiddenState​\(fϕ​\(ci\)\)\)\\mathbf\{e\}\_\{i\}\\leftarrow\\text\{MeanPool\}\(\\text\{LastHiddenState\}\(f\_\{\\phi\}\(c\_\{i\}\)\)\)⊳\\trianglerightEq\.[3](https://arxiv.org/html/2607.03397#S4.E3)

6:

𝐞^i←𝐞i/‖𝐞i‖2\\hat\{\\mathbf\{e\}\}\_\{i\}\\leftarrow\\mathbf\{e\}\_\{i\}\\,/\\,\\\|\\mathbf\{e\}\_\{i\}\\\|\_\{2\}
7:endfor

8:Assemble

E^←\[𝐞^1,…,𝐞^n\]⊤\\hat\{E\}\\leftarrow\[\\hat\{\\mathbf\{e\}\}\_\{1\},\\ldots,\\hat\{\\mathbf\{e\}\}\_\{n\}\]^\{\\top\}
9:

𝐀TE←E^​E^⊤\\mathbf\{A\}^\{\\mathrm\{TE\}\}\\leftarrow\\hat\{E\}\\,\\hat\{E\}^\{\\top\}⊳\\trianglerightAi​jTE=𝐞^i⊤​𝐞^jA^\{\\mathrm\{TE\}\}\_\{ij\}=\\hat\{\\mathbf\{e\}\}\_\{i\}^\{\\top\}\\hat\{\\mathbf\{e\}\}\_\{j\}, Eq\.[4](https://arxiv.org/html/2607.03397#S4.E4)

10:return

𝐀TE\\mathbf\{A\}^\{\\mathrm\{TE\}\}

Algorithm 2Denoiser Concept Graph Extraction1:Concept set

𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}, denoiser

ϵθ\\epsilon\_\{\\theta\}with

LLcross\-attention layers \(

HHheads each\), sensitivity

ξ\\xi
2:Adjacency matrix

𝐀𝒰∈ℝn×n\\mathbf\{A\}^\{\\mathcal\{U\}\}\\in\\mathbb\{R\}^\{n\\times n\}
3:for

u=1u=1to

nndo

4:Generate image

IcuI\_\{c\_\{u\}\}by running the full denoising chain conditioned on

cuc\_\{u\}
5:Compute CLIP alignment loss

ℒcu=CLIPScore​\(Icu,cu\)\\mathcal\{L\}\_\{c\_\{u\}\}=\\mathrm\{CLIPScore\}\(I\_\{c\_\{u\}\},c\_\{u\}\)⊳\\trianglerightEq\.[1](https://arxiv.org/html/2607.03397#S4.E1)

6:for

r∈\{k,q,v\}r\\in\\\{k,q,v\\\}do

7:for

ℓ=1\\ell=1to

LL,

h=1h=1to

HHdo

8:

Grℓ,h​\(cu\)←mean​\(\|∇Wrℓ,hℒcu\|\)G^\{\\ell,h\}\_\{r\}\(c\_\{u\}\)\\leftarrow\\mathrm\{mean\}\\\!\\left\(\|\\nabla\_\{W^\{\\ell,h\}\_\{r\}\}\\mathcal\{L\}\_\{c\_\{u\}\}\|\\right\)⊳\\trianglerightEq\.[5](https://arxiv.org/html/2607.03397#S4.E5)

9:endfor

10:Flatten:

𝐆r​\(cu\)←\[Gr1,1​\(cu\),…,GrL,H​\(cu\)\]⊤\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\\leftarrow\[G^\{1,1\}\_\{r\}\(c\_\{u\}\),\\ldots,G^\{L,H\}\_\{r\}\(c\_\{u\}\)\]^\{\\top\}
11:

γr←ξ⋅σ​\(𝐆r​\(cu\)\)\+μ​\(𝐆r​\(cu\)\)\\gamma\_\{r\}\\leftarrow\\xi\\cdot\\sigma\(\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\)\+\\mu\(\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\)⊳\\trianglerightAdaptive threshold, Eq\.[6](https://arxiv.org/html/2607.03397#S4.E6)

12:

ℳr​\(cu\)←𝟏​\[𝐆r​\(cu\)\>γr\]\\mathcal\{M\}\_\{r\}\(c\_\{u\}\)\\leftarrow\\mathbf\{1\}\[\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\>\\gamma\_\{r\}\]
13:

𝐠r​\(cu\)←𝐆r​\(cu\)⊙ℳr​\(cu\)\\mathbf\{g\}\_\{r\}\(c\_\{u\}\)\\leftarrow\\mathbf\{G\}\_\{r\}\(c\_\{u\}\)\\odot\\mathcal\{M\}\_\{r\}\(c\_\{u\}\)⊳\\trianglerightFingerprint, Eq\.[7](https://arxiv.org/html/2607.03397#S4.E7)

14:endfor

15:endfor

16:for

i=1i=1to

nn,

j=1j=1to

nndo

17:

Ai​j𝒰←13​∑r∈\{k,q,v\}cos⁡\(𝐠r​\(ci\),𝐠r​\(cj\)\)A^\{\\mathcal\{U\}\}\_\{ij\}\\leftarrow\\dfrac\{1\}\{3\}\\displaystyle\\sum\_\{r\\in\\\{k,q,v\\\}\}\\cos\\\!\\left\(\\mathbf\{g\}\_\{r\}\(c\_\{i\}\),\\;\\mathbf\{g\}\_\{r\}\(c\_\{j\}\)\\right\)⊳\\trianglerightEq\.[8](https://arxiv.org/html/2607.03397#S4.E8)

18:endfor

19:return

𝐀𝒰\\mathbf\{A\}^\{\\mathcal\{U\}\}

Algorithm 3Concept Graph Alignment \(Text Encoder or Denoiser\)1:Concept set

𝒞\\mathcal\{C\}, target graph

𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}, supervised rows

𝒮\\mathcal\{S\}, anchor concepts

𝒞S\\mathcal\{C\}\_\{S\}, anchor weights

\{λc\}c∈𝒞S\\\{\\lambda\_\{c\}\\\}\_\{c\\in\\mathcal\{C\}\_\{S\}\}, loss weights

δ,γ\\delta,\\gamma, temperature

τ\\tau, training steps

TT
2:Aligned model parameters

θ∗\\theta^\{\*\}
3:Freeze parameters of the component*not*being aligned

4:Capture original anchor representations:

5:for

c∈𝒞Sc\\in\\mathcal\{C\}\_\{S\}do

𝐚c\(0\)←\\mathbf\{a\}^\{\(0\)\}\_\{c\}\\leftarrowcurrent embedding or CG\-row of

cc
6:endfor

7:for

t=1t=1to

TTdo

8:Extract current CG

𝐀\\mathbf\{A\}\(Alg\.[1](https://arxiv.org/html/2607.03397#alg1)or[2](https://arxiv.org/html/2607.03397#alg2)\) retaining the computation graph

9:for

i∈𝒮i\\in\\mathcal\{S\},

j=1j=1to

nndo

10:Compute soft rank

r~i​j←1n​\(1\+∑kσ​\(Ai​k−Ai​jτ\)\)\\tilde\{r\}\_\{ij\}\\leftarrow\\tfrac\{1\}\{n\}\\\!\\left\(1\+\\sum\_\{k\}\\sigma\\\!\\left\(\\tfrac\{A\_\{ik\}\-A\_\{ij\}\}\{\\tau\}\\right\)\\\!\\right\)⊳\\trianglerightEq\.[10](https://arxiv.org/html/2607.03397#S5.E10)

11:Compute hard rank target

r¯i​jGT\\bar\{r\}^\{\\mathrm\{GT\}\}\_\{ij\}from

𝐀GT\\mathbf\{A\}^\{\\mathrm\{GT\}\}with average\-rank tie\-breaking⊳\\trianglerightEq\.[11](https://arxiv.org/html/2607.03397#S5.E11)

12:endfor

13:

ℒrank←1\|𝒮\|⋅n​∑i∈𝒮∑j\(r~i​j−r¯i​jGT\)2\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\\leftarrow\\dfrac\{1\}\{\|\\mathcal\{S\}\|\\cdot n\}\\displaystyle\\sum\_\{i\\in\\mathcal\{S\}\}\\sum\_\{j\}\\\!\\left\(\\tilde\{r\}\_\{ij\}\-\\bar\{r\}^\{\\mathrm\{GT\}\}\_\{ij\}\\right\)^\{\\\!2\}⊳\\trianglerightEq\.[12](https://arxiv.org/html/2607.03397#S5.E12)

14:

ℒanchor←∑c∈𝒞Sλc​‖𝐚c\(0\)−𝐚c‖2\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\\leftarrow\\displaystyle\\sum\_\{c\\in\\mathcal\{C\}\_\{S\}\}\\lambda\_\{c\}\\,\\\|\\mathbf\{a\}^\{\(0\)\}\_\{c\}\-\\mathbf\{a\}\_\{c\}\\\|^\{2\}⊳\\trianglerightEq\.[13](https://arxiv.org/html/2607.03397#S5.E13)or[14](https://arxiv.org/html/2607.03397#S5.E14)

15:

ℒpres←𝔼\(x0,c~\)∼𝒟pres,ϵ,t​\[‖ϵ−ϵθ​\(zt,fϕ​\(c~\),t\)‖2\]\\mathcal\{L\}\_\{\\mathrm\{pres\}\}\\leftarrow\\mathbb\{E\}\_\{\(x\_\{0\},\\tilde\{c\}\)\\sim\\mathcal\{D\}\_\{\\mathrm\{pres\}\},\\,\\epsilon,\\,t\}\\\!\\left\[\\\|\\epsilon\-\\epsilon\_\{\\theta\}\(z\_\{t\},f\_\{\\phi\}\(\\tilde\{c\}\),t\)\\\|^\{2\}\\right\]⊳\\trianglerightEq\.[15](https://arxiv.org/html/2607.03397#S5.E15)

16:

ℒ←δ⋅ℒrank\+γ⋅ℒpres\+ℒanchor\\mathcal\{L\}\\leftarrow\\delta\\cdot\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\+\\gamma\\cdot\\mathcal\{L\}\_\{\\mathrm\{pres\}\}\+\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}⊳\\trianglerightEq\.[16](https://arxiv.org/html/2607.03397#S5.E16)

17:

θ←θ−η​∇θℒ\\theta\\leftarrow\\theta\-\\eta\\,\\nabla\_\{\\theta\}\\,\\mathcal\{L\}
18:endfor

19:return

θ\\theta

## Appendix CBackground on T2I Diffusion Models

T2I Diffusion ModelsLatent diffusion models\[Rombachet al\.,[2022](https://arxiv.org/html/2607.03397#bib.bib35)\]operate by learning to invert a fixed forward noising process\. Given a data samplex0x\_\{0\}, a forward process corrupts it into Gaussian noisexTx\_\{T\}overTTtimesteps via:

q​\(xt\|xt−1\):=𝒩​\(xt;αt​xt−1,\(1−αt\)​𝐈\),q\(x\_\{t\}\|x\_\{t\-1\}\):=\\mathcal\{N\}\\\!\\left\(x\_\{t\};\\,\\sqrt\{\\alpha\_\{t\}\}\\,x\_\{t\-1\},\\,\(1\-\\alpha\_\{t\}\)\\mathbf\{I\}\\right\),\(18\)whereαt∈\(0,1\)\\alpha\_\{t\}\\in\(0,1\)is the variance schedule\. A denoiserϵθ\\epsilon\_\{\\theta\}, parameterised as a conditional UNet, is trained to predict the added noise:

ℒLDM=𝔼zt,ϵ,t,c​\[‖ϵ−ϵθ​\(zt,c,t\)‖2\],\\mathcal\{L\}\_\{\\text\{LDM\}\}=\\mathbb\{E\}\_\{z\_\{t\},\\epsilon,t,c\}\\left\[\\\|\\epsilon\-\\epsilon\_\{\\theta\}\(z\_\{t\},c,t\)\\\|^\{2\}\\right\],\(19\)whereztz\_\{t\}is the noisy latent at timestepttandccis a text embedding obtained from a pretrained CLIP text encoder\[Radfordet al\.,[2021](https://arxiv.org/html/2607.03397#bib.bib47)\]\.

Cross\-Attention\.Semantic alignment between text and image is mediated by cross\-attention \(CA\) at each layer of the denoiser\. Given image\-derived queriesQ∈ℝN×dQ\\in\\mathbb\{R\}^\{N\\times d\}and text\-derived keysK∈ℝL×dK\\in\\mathbb\{R\}^\{L\\times d\}, valuesV∈ℝL×dV\\in\\mathbb\{R\}^\{L\\times d\}:

Cross Attention​\(Q,K,V\)=softmax​\(Q​K⊤d\)​V,\\text\{Cross Attention\}\(Q,K,V\)=\\text\{softmax\}\\\!\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)V,\(20\)whereNNis the number of spatial locations inztz\_\{t\}andLLis the number of text tokens\.KKandVVare linear projections of the text embeddingcc, making the CA layers the primary site of text\-visual concept binding\[Orgadet al\.,[2023](https://arxiv.org/html/2607.03397#bib.bib13), Gandikotaet al\.,[2024](https://arxiv.org/html/2607.03397#bib.bib14)\]\. The projection matricesWKW\_\{K\}andWVW\_\{V\}therefore encode the model’s concept representations and are the principal target of both bias mitigation and concept unlearning methods\.

## Appendix DEvaluation Metrics

We use the following metrics for the bias mitigation accessment:

- •Fairness Discrepancy \(FD\)Pariharet al\.\[[2024](https://arxiv.org/html/2607.03397#bib.bib16)\]: deviation of the predicted demographic distribution from the uniform target,FD=∑i\(p¯i−1/K\)2\\mathrm\{FD\}=\\sqrt\{\\sum\_\{i\}\(\\bar\{p\}\_\{i\}\-1/K\)^\{2\}\}, wherep¯i\\bar\{p\}\_\{i\}is the mean predicted probability for classiiandKKis the number of classes\. Lower is fairer\.
- •FIDHeuselet al\.\[[2017](https://arxiv.org/html/2607.03397#bib.bib55)\]: Fréchet Inception Distance against FFHQ as the reference distribution\. Lower is better\.
- •CLIP\-I\(image fidelity\): mean cosine similarity between CLIPRadfordet al\.\[[2021](https://arxiv.org/html/2607.03397#bib.bib47)\]image embeddings of generated and base\-model reference images, normalised to\[0,1\]\[0,1\]\. Higher indicates better content preservation\.
- •CLIP\-T\(text alignment\): mean cosine similarity between CLIP image and text embeddings for the generation prompt, normalised to\[0,1\]\[0,1\]\. Higher indicates better semantic fidelity to the prompt\.

## Appendix EReproducibility Statement: Hyperparameters, Implementation Details and Compute

### Implementation details\.

Both stages ofCO\-ALIGNuse AdamW with learning rateη=10−5\\eta=10^\{\-5\}, trained for33epochs of300300steps each with batch size44\. The ranking loss temperature isτ=0\.05\\tau=0\.05and the ranking loss weight isδ=1\.0\\delta=1\.0; the preservation loss is disabled during alignment as the static anchor penalty provides sufficient regularisation against representation collapse\. Anchor concept weights are set uniformly toλc=1\.0\\lambda\_\{c\}=1\.0for allc∈𝒞Sc\\in\\mathcal\{C\}\_\{S\}\. The concept neuron sensitivity threshold usesξ=2\.0\\xi=2\.0\(i\.e\. two standard deviations above the mean gradient magnitude\)\.

Text encoder stage\.We inject LoRA adapters into the CLIP text encoder’s\{Q,K,V,out\}\\\{Q,K,V,\\mathrm\{out\}\\\}projection matrices with rankr=8r=8and scalingα=16\\alpha=16; all other text encoder parameters are frozen\. Concept neurons are identified using the CLIP\-score lossℒcu\\mathcal\{L\}\_\{c\_\{u\}\}\.

Denoiser stage\.The UNet’s cross\-attention\{K,Q,V\}\\\{K,Q,V\\\}projection weights are trained; all other parameters \(including the text encoder and VAE\) are frozen\. The denoiser concept graph is computed via the accumulate\-all\-steps differentiable gradient scheme: at each training iteration,55denoising timesteps are sampled uniformly from the3535\-step DDIM trajectory, and the ranking loss gradient is accumulated over the sampled steps with a fresh random latent seed per iteration\. Static concept graph rows are anchored at their pre\-alignment values withλc=1\.0\\lambda\_\{c\}=1\.0\.

All experiments are run on a single NVIDIA A100 80GB GPU\. The wall clock time for aligning the text encoder is roughly 5 minutes\. The wall clock time for aligning the denoiser\(UNet\) is 3 hours\.

## Appendix FVLM\-based evaluation for adversarial robustness

Two of our evaluation tasks require open\-ended image understanding that rule\-based classifiers cannot provide: measuring theincoherence rate\(whether a generated image depicts the intended target concept\) and measuringnudity bypassunder adversarial prompts \(whether a generated image contains harmful content\)\. For both tasks we use LLaVA\-v1\.6\-Mistral\-7BLiuet al\.\[[2024](https://arxiv.org/html/2607.03397#bib.bib62)\]\(llava\-hf/llava\-v1\.6\-mistral\-7b\-hf\) as a zero\-shot visual classifier, querying it with a structured natural\-language prompt for each generated image\. Generation uses3030DDIM steps with guidance scale7\.57\.5; VLM decoding is greedy \(do\_sample=False\)\.

Incoherence rate\.For a generated image produced with promptpp\(e\.g\.“Nurse”\), we ask the VLM to classify whether the image depicts the target concept or one of its attribute variants𝒞check\\mathcal\{C\}\_\{\\mathrm\{check\}\}\(e\.g\. \{“Male Nurse”, “Female Nurse”\}\)\. The prompt template is:

> This image was generated with the prompt “\{target\}”\. Which of the following best describes what is shown:\{options\}, or None? Answer with exactly one of:\{options\}, None\.

The VLM response is matched to the closest concept in𝒞check\\mathcal\{C\}\_\{\\mathrm\{check\}\}via longest\-first substring matching; images that matchNoneare counted as incoherent\. The incoherence rate is the fraction of images assignedNoneacross all generated images for a given concept\.

Nudity detection\.For each adversarial conceptc∈𝒞advc\\in\\mathcal\{C\}\_\{\\mathrm\{adv\}\}\(e\.g\.“Nymphettes”,“Creampie”\), we ask the VLM a binary question for each generated image:

> Is there\{concept\}in this image? Answer only yes or no\.

An image is flagged if the response begins with“yes”\. The reported score is the percentage of flagged images per concept, averaged as shown in Table[3](https://arxiv.org/html/2607.03397#S7.T3)\. Using a VLM rather than a nudity\-specific classifier is deliberate: adversarial bypass concepts such asCreampieorNymphettesare semantically indirect, and a VLM with broad visual understanding is better positioned to judge whether the generated content is harmful than a classifier trained only on explicit nudity labels\.

## Appendix GQualitative Results

![Refer to caption](https://arxiv.org/html/2607.03397v1/x7.png)Figure 7:This figure shows some qualitative results of the debiasing across gender, race and age for the professions of Doctor and Engineer\. The images are sampled during theCO\-ALIGN’s procedure\. Observe the changes from left to right\.Figure[7](https://arxiv.org/html/2607.03397#A7.F7)shows debiasing usingCO\-ALIGNin the three paradigms of gender, race and age on the professions of nurse, doctor and engineer respectively\.

### Limitations\.

CO\-ALIGNhas two principal limitations\. First, extracting the denoiser concept graph requires one full denoising trajectory per concept per training step, making UNet alignment more compute\-intensive than text\-encoder\-only methods\. Second, our experiments are conducted on Stable Diffusion v1\.5; extension to DiT\-based or SDXL architectures, where the text\-image coupling geometry differs, requires re\-deriving the concept graph extraction for transformer\-based denoisers, which we leave to future work\.

## 9\. Licenses for Existing Assets

All assets used in this work are listed below with their respective licenses\.

- •Stable Diffusion v1\.5Rombachet al\.\[[2022](https://arxiv.org/html/2607.03397#bib.bib35)\]\(runwayml/stable\-diffusion\-v1\-5\): CreativeML Open RAIL\-M License\. We use this model as the base generative model for all experiments\.
- •Diffusers libraryvon Platen and others \[[2022](https://arxiv.org/html/2607.03397#bib.bib63)\]: Apache License 2\.0\. Used for model loading, inference pipelines, and scheduler implementations\.
- •CLIPRadfordet al\.\[[2021](https://arxiv.org/html/2607.03397#bib.bib47)\]: MIT License \(OpenAI\)\. Used as the text encoder within Stable Diffusion and for CLIP\-I/CLIP\-T metric computation\.
- •FairFace classifierKärkkäinen and Joo \[[2021](https://arxiv.org/html/2607.03397#bib.bib52)\]: CC BY 4\.0\. Used for demographic classification of generated faces across gender, race, and age axes\.
- •LLaVA\-v1\.6\-Mistral\-7BLiuet al\.\[[2024](https://arxiv.org/html/2607.03397#bib.bib62)\]: Apache License 2\.0\. Used as a zero\-shot VLM classifier for incoherence rate and nudity bypass evaluation\.
- •WordNet\(Princeton\): Princeton WordNet License \(free for research use\)\. Used to derive the ground\-truth knowledge graph for general concept editing experiments\.
- •UCEGandikotaet al\.\[[2024](https://arxiv.org/html/2607.03397#bib.bib14)\]andESDGandikotaet al\.\[[2023](https://arxiv.org/html/2607.03397#bib.bib56)\]: MIT License\. Used as post\-hoc unlearning baselines\.
- •SAFREEYoonet al\.\[[2025](https://arxiv.org/html/2607.03397#bib.bib57)\]: MIT License\. Used as a post\-hoc unlearning baseline\.
- •Debias\-VLChuanget al\.\[[2023](https://arxiv.org/html/2607.03397#bib.bib11)\]: MIT License\. Used as a text\-encoder debiasing baseline\.
- •Finetuning baselineShenet al\.\[[2024](https://arxiv.org/html/2607.03397#bib.bib23)\]: MIT License\. Used as a text\-encoder debiasing baseline\.
- •MS\-COCOLin and others \[[2014](https://arxiv.org/html/2607.03397#bib.bib64)\]: CC BY 4\.0\. A subset of image\-caption pairs is used as the preservation reference dataset𝒟pres\\mathcal\{D\}\_\{\\mathrm\{pres\}\}during alignment\.

## 10\. Safeguards

CO\-ALIGN edits the internal concept ontology of a generative model, which carries dual\-use risk: the same mechanism that reduces demographic bias could in principle be used to amplify it, or to steer concept associations in harmful directions\. We take the following precautions\.

Release\.We will release model checkpoints only for the debiased variants reported in the paper, together with the alignment configurations used to produce them\. We do not release checkpoints from intermediate experiments or from the unlearning pre\-alignment stage in isolation, as these represent partially edited models whose behaviour is harder to audit\.

Scope of edits\.All released configurations target occupational demographic bias \(gender, race, age\) and nudity unlearning, both well\-studied, socially motivated editing objectives with established evaluation protocols\. We do not release configurations that could be adapted to amplify bias or generate targeted harmful content\.

Misuse of neighbourhood propagation\.The neighbourhood pulling effect \(§[3](https://arxiv.org/html/2607.03397#S7.T3)\) is an emergent property that could in principle be exploited to covertly shift concept associations beyond the declared edit scope\. We document this effect explicitly so that auditors of CO\-ALIGN\-edited models can monitor the broader semantic neighbourhood of any edited concept, not only the directly supervised concepts\.

Intended use\.CO\-ALIGN is intended for researchers and practitioners seeking to reduce demographic bias in deployed T2I models or to improve the robustness of concept unlearning\. Use of the method to produce, distribute, or enable the generation of content that is illegal, harmful, or discriminatory is explicitly out of scope and contrary to its design intent\.

## 11\. Broader Impacts

### Positive impacts\.

Demographic bias in large\-scale T2I models has documented societal consequences, reinforcing stereotypes at the scale of internet\-wide deployment\. CO\-ALIGN offers a practical route to post\-hoc correction without retraining from scratch, lowering the barrier for practitioners to deploy fairer models\. The mechanistic framing, bias as asymmetry in concept graph topology rather than as a property of individual parameters, also contributes to interpretability research by providing a structured diagnostic tool for auditing a model’s internal concept associations before deployment\. The unlearning application further demonstrates that the same framework can strengthen existing safety mechanisms against adversarial prompt attacks, with implications for responsible deployment of generative models in consumer\-facing systems\.

### Negative impacts and limitations\.

As noted in limitations, CO\-ALIGN currently targets Stable Diffusion v1\.5 and has not been validated on more recent architectures\. Practitioners applying the method to other models should re\-evaluate both the fairness gains and the incoherence trade\-off before deployment\. More broadly, no post\-hoc bias mitigation technique, including CO\-ALIGN, can fully compensate for the scale and variety of biases present in web\-scraped training corpora; residual bias will remain, and the corrected model should not be treated as fully unbiased\. Finally, the dual\-use risk of concept graph editing \(discussed in §10\) warrants ongoing monitoring of deployed edited models, particularly the semantic neighbourhood of any edited concept, to detect unintended drift in adjacent concept associations\.