Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? -- A Theoretical and Empirical Study

# Do Counterfactually Fair Image Classifiers Satisfy Group Fairness? – A Theoretical and Empirical Study
Source: [https://arxiv.org/html/2607.06603](https://arxiv.org/html/2607.06603)
Sangwon Jung1Sumin Yu111footnotemark:1Sanghyuk Chun2Taesup Moon1,322footnotemark:2 1Department of Electrical and Computer Engineering, Seoul National University 2NAVER AI Lab3ASRI/INMC/IPAI/AIIS, Seoul National University

###### Abstract

The notion of algorithmic fairness has been actively explored from various aspects of fairness, such as counterfactual fairness \(CF\) and group fairness \(GF\)\. However, the exact relationship between CF and GF remains to be unclear, especially in image classification tasks; the reason is because we often cannot collect counterfactual samples regarding a sensitive attribute, essential for evaluating CF, from the existing images \(*e\.g\.*, a photo of the same person but with different secondary sex characteristics\)\. In this paper, we construct new image datasets for evaluating CF by using a high\-quality image editing method and carefully labeling with human annotators\. Our datasets, CelebA\-CF and LFW\-CF, build upon the popular image GF benchmarks; hence, we can evaluate CF and GF simultaneously\. We empirically observe that CF does not imply GF in image classification, whereas previous studies on tabular datasets observed the opposite\. We theoretically show that it could be due to the existence of a latent attributeGGthat is correlated with, but not caused by, the sensitive attribute \(*e\.g\.*, secondary sex characteristics are highly correlated with hair length\)\. From this observation, we propose a simple baseline, Counterfactual Knowledge Distillation \(CKD\), to mitigate such correlation with the sensitive attributes\. Extensive experimental results on CelebA\-CF and LFW\-CF demonstrate that CF\-achieving models satisfy GF if we successfully reduce the reliance onGG\(*e\.g\.*, using CKD\)\.

## 1Introduction

As machine learning algorithms are deployed in societal computer vision applications such as facial recognition\[[39](https://arxiv.org/html/2607.06603#bib.bib73)\]and job interview\[[29](https://arxiv.org/html/2607.06603#bib.bib69)\], concerns have grown regarding their potential to discriminate against certain individuals and groups\. For instance, a face recognition system might exhibit disparate accuracies across different demographic groups\[[3](https://arxiv.org/html/2607.06603#bib.bib72)\], while a job interview algorithm could be biased based on protective attributes even for the same interviewee\[[11](https://arxiv.org/html/2607.06603#bib.bib163)\]\. Consequently,algorithmic fairnessin image classifiers has gained significant attention in academic and industrial research communities\.

While conceptually apparent, determining a concrete notion of fairness is challenging, leading to the proposal of several different fairness notions\. One prevalent notion iscounterfactual fairness\(CF\)\[[23](https://arxiv.org/html/2607.06603#bib.bib131)\]which seeks consistent predictions when only a sensitive attribute is intervened\. Another important notion isgroup fairness\(GF\)\[[43](https://arxiv.org/html/2607.06603#bib.bib83)\]that aims to treat different demographic groups equally to prevent one group unfairly disadvantaged compared to another\. Many researchers have focused on developing separate algorithms to achieve each notion, while understanding the exact relationship between CF and GF is yet under\-explored;e\.g\.,some recent work\[[1](https://arxiv.org/html/2607.06603#bib.bib51),[35](https://arxiv.org/html/2607.06603#bib.bib9)\]showed that a model achieving CF can meet several GF notionsonlyunder specific conditions of Structural Causal Models\.

Furthermore, previous studies on the relationship between CF and GF have not considered the setting of image classification due to the absence ofevaluationdatasets with counterfactual images, in which only the sensitive attribute is altered from the original images while other attributes not caused by the sensitive attribute remain unchanged — a data nearly impossible to collect in the real world\. There have been several works generating counterfactual images using generative models\[[5](https://arxiv.org/html/2607.06603#bib.bib49),[19](https://arxiv.org/html/2607.06603#bib.bib59),[26](https://arxiv.org/html/2607.06603#bib.bib7),[32](https://arxiv.org/html/2607.06603#bib.bib6),[44](https://arxiv.org/html/2607.06603#bib.bib65),[34](https://arxiv.org/html/2607.06603#bib.bib62),[4](https://arxiv.org/html/2607.06603#bib.bib66)\], but they have only focused on utilizing generated counterfactual samples for training rather than evaluation\. Moreover, these methods often suffer from low\-quality counterfactual images generated based on VAE\[[20](https://arxiv.org/html/2607.06603#bib.bib30)\]or GAN\[[9](https://arxiv.org/html/2607.06603#bib.bib31)\]\. One notable exception isLianget al\.\[[24](https://arxiv.org/html/2607.06603#bib.bib36)\], which offers an evaluation dataset including counterfactual images\. However, their images are all synthetic; thus, it is still insufficient to evaluate CF due to distribution shifts from real\-world images\.

In this paper, we construct CF benchmarks for image classification tasks using high\-performing diffusion model\-based generative models\. Our datasets build upon popular facial benchmark datasets used for evaluating GF, CelebA and LFW, by altering the sensitive attribute with pre\-trained InstructPix2Pix \(IP2P\)\[[2](https://arxiv.org/html/2607.06603#bib.bib26)\]\. We then carefully curate the edited samples by human annotators and verify the reliability of our datasets as counterfactual samples from additional annotators\. Note that our datasets, CelebA\-Counterfactual Face \(CelebA\-CF\) and LFW\-Counterfactual Face \(LFW\-CF\), share the same test samples as the original GF benchmarks, enabling the evaluation of both GF and CF\.

Using our datasets, we conduct a primitive study on the relationship between CF and GF in image classification,*e\.g\.*, test whether CF implies GF for image classifiers\. To that end, we train CF\-aware methods\[[36](https://arxiv.org/html/2607.06603#bib.bib8),[7](https://arxiv.org/html/2607.06603#bib.bib176)\]and evaluate them with our datasets using both CF and GF metrics\. From the result, we observe that they achieve CF but fail to satisfy GF, contrary to previous findings that CF can imply GF\[[1](https://arxiv.org/html/2607.06603#bib.bib51),[35](https://arxiv.org/html/2607.06603#bib.bib9)\]\. We attribute this failure to Structural Causal Models \(SCMs\) of image generation\. Specifically, for an image SCM, a latent attributeGGis more likely to exist, which could be correlated with, but not caused by, the sensitive attributes\. For example, in the case where the sensitive attribute is the sex of a person in an image, secondary sex characteristics such as beard and hairline are highly correlated with hair length, but it does not mean that such characteristics cause the length of hair\. In this scenario, if a model achieving CF relies on the attributeGG\(*e\.g\.*, hair length\) on its prediction, it could more severely violate GF in the worst case\. Therefore, if we can reduce the dependency onGGof a CF\-aware model, we may achieve both CF and GF\. Empirically, we find that a model trained with vanilla cross\-entropy loss is more robust toGGthan a model trained with a CF\-aware method\. Motivated by this, we propose a simple baseline, named Counterfactual Knowledge Distillation \(CKD\), which distills the robustness toGGduring the original CF\-aware optimization\. Finally, our extensive experiments using CelebA\-CF and LFW\-CF demonstrate that CF\-achieving models satisfy GF when reducing the reliance onGG\(*e\.g\.*, using CKD\)\.

In summary, our contributions are three\-fold\. Firstly, we construct two new image classification benchmarks for measuring CF, CelebA\-CF and LFW\-CF\. Secondly, using these datasets, we observe the disparity between CF and GF in image classifiers and provide a theoretical rationale; a counterfactually fair classifier may not necessarily achieve GF when an additional latent attribute that is correlated with the sensitive attribute exists\. Finally, we propose a simple baseline, CKD, to reduce the sensitivity to such latent attributes of a model, resulting in achieving CF and GF simultaneously\.

## 2Constructing high\-quality counterfactual images

The degree of counterfactual fairness \(CF\) can be measured by the prediction consistency between an original sample and its corresponding counterfactual \(CTF\) sample\. For a given sample and a sensitive attribute, a CTF sample is defined as the one of which the sensitive attribute is altered while all the other attributes not caused by the sensitive attribute remain the same\. However, acquiring a CTF sample for an image is challenging\. For example, if the sex of a person in an image is the sensitive attribute, obtaining a CTF sample requires changing the secondary sex characteristics of the person such as beard or hairline, while preserving their identity and the other attributes, which is impossible in practice\. One possible alternative is to generate a virtual face by altering such secondary sex characteristics of the given identity using a high\-quality image editing method\.

Several previous approaches\[[19](https://arxiv.org/html/2607.06603#bib.bib59),[34](https://arxiv.org/html/2607.06603#bib.bib62),[41](https://arxiv.org/html/2607.06603#bib.bib64),[44](https://arxiv.org/html/2607.06603#bib.bib65),[22](https://arxiv.org/html/2607.06603#bib.bib50)\]have attempted to generate CTF images by VAE or GAN\-based editing methods\. However, they have struggled with low image quality or unintended modifications to non\-sensitive attributes, rendering them unreliable for evaluating CF\. To address such issues, we employ IP2P\[[2](https://arxiv.org/html/2607.06603#bib.bib26)\], an advanced diffusion model\-based image editing method\. Notably, IP2P can generate high\-quality CTF samples by simply adjusting the text instructions without any model retraining\.

As the first step, we edit the test images of two popular facial image datasets, CelebA\[[25](https://arxiv.org/html/2607.06603#bib.bib98)\]and LFW\[[13](https://arxiv.org/html/2607.06603#bib.bib20)\]\. We choose the “sex” of a person in an image as the sensitive attribute111The two datasets use the terms “gender” for indicating their sensitive attributes\. However, using such terminology can present some ethical concerns because they can suggest meanings linked to social identities\. Thus, we have decided to use the term “sex” instead, which more accurately refers to biological characteristics\.and edit the sex\-related visual characteristics of facial images using text prompts\. We generated 720 CelebA CTF image pairs and 632 LFW CTF image pairs, where the images are selected to be balanced across groups for both target and sensitive labels\. Here, we treat “blond hair” and “smiling” as the target labels for CelebA and LFW, respectively\. Namely, for example, the CelebA CTF image pairs have a balanced group of <female, non\-blond hair\>, …, and so on\.[Figure˜1](https://arxiv.org/html/2607.06603#S2.F1)and[A\.1](https://arxiv.org/html/2607.06603#A1.F1)show examples of generated CTF images together with the originals\. Hyperparameter settings are reported in[Section˜C\.1](https://arxiv.org/html/2607.06603#A3.SS1)\. Note that while we adopt the “sex” attribute, our generation process is attribute\-agnostic \(*e\.g\.*, age or skin color can be also used in place of sex\) as illustrated in[Figure˜A\.2](https://arxiv.org/html/2607.06603#A1.F2)\.

![Refer to caption](https://arxiv.org/html/2607.06603v1/x1.png)Figure 1:CelebA\-CF examples\. The counterfactual \(CTF\) images regarding the “sex attribute” are shown\.Image filtering\. Despite the high quality of IP2P, low\-quality CTF images can still be generated\. To address this, we employed five human annotators to filter the images,*i\.e\.*, each image pair was annotated as either “correct” or “incorrect”\. To ensure objective and precise annotation criteria, we created guidelines as follows\. Initially, we compiled a list of 20 masculine and feminine visual features using GPT\-4o and with guidance from experts specialized in fairness, selected nine key facial attributes representing sex\-related visual characteristics: facial hair, Adam’s apple, skin texture, jawline, chin shape, brow ridge, cheekbone prominence, lip fullness, and hairline\. These attributes were used to establish the criteria for evaluating correct CTF samples\. One notable issue is that most of the feminine\-like images in CelebA and LFW datasets include makeups \(for instance, many female celebrities in the CelebA dataset appear to be wearing makeup\) and the IP2P model is biased towards removing makeup when altering feminine features\. To prevent images from being filtered out solely due to changes in makeup, we additionally included makeup in the set of key attributes, even though it is not a sex characteristic\. Finally, the guidelines were created based on these ten attributes, providing some criteria for correct CTF samples, such as whether the change of some of the ten attributes was accurate and whether other facial characteristics remained consistent with the original image\. Using these guidelines, we filtered out pairs receiving two or fewer “correct” votes, resulting in 230 and 144 images for CelebA and LFW, respectively\. More details about the human annotating interface are in[Section˜C\.2](https://arxiv.org/html/2607.06603#A3.SS2), and additional information on the newly created dataset can be found in[Section˜C\.4](https://arxiv.org/html/2607.06603#A3.SS4)\.

Table 1:Human evaluation of the reliability of our datasets\. Accuracies of the correctly altered sensitive attributes and well\-preserved non\-sensitive attributes are shown\.Reliability check\. We further verify the quality of our datasets by additional five human annotators, distinct from those participated in the filtering process\. Those annotators evaluate only the images that remained after the filtering, based on two criteria: \(1\) whether the sensitive attribute was correctly changed and \(2\) whether the other non\-sensitive attributes were preserved\. The annotators evaluated the images for the sensitive attribute, “sex” and three non\-sensitive attributes, “blond hair”, “gray hair”, and “smiling”; we chose these three because other attributes can be subjective \(*e\.g\.*, “big nose”\)\[[40](https://arxiv.org/html/2607.06603#bib.bib34)\]or had already been filtered \(*e\.g\.*, “wearing hat”\)\. Details of the annotating interface provided to the five annotators are in[Section˜C\.2](https://arxiv.org/html/2607.06603#A3.SS2)\. Based on the majority vote, we compute the percentage of CTF samples which met each of the two criteria,*i\.e\.*, the accuracies for whether the sensitive and non\-sensitive attributes are correctly altered and preserved\.[Table˜1](https://arxiv.org/html/2607.06603#S2.T1)displays the values for CelebA\-CF and LFW\-CF\. The non\-sensitive accuracy is averaged across three non\-sensitive attributes\. The results demonstrate that our CTF samples almost meet the two CTF criteria, suggesting that our datasets can be reliably utilized to evaluate CF\.

Ethical considerations\. In our study, we use the term “sex”, not “gender”, to represent the sensitive attribute with biological traits, because terms such as “gender” might imply associations with social identities, potentially raising some ethical issues\. We also specifically choose ten perceived facial attributes as the visual features representing the biological sex in facial images\. We believe that these considerations help alleviate various normative harms that arise from dichotomizing gender, which refers to social identity\. However, despite our efforts, the sex\-related visual characteristics are complex and intertwined, making it challenging to fully represent with a binary label\. Thus, we urge practitioners to use our datasets with these considerations in mind\.

## 3Primitive study on the relationship between CF and GF

### 3\.1Experimental setup

We consider the image classification task where each data sample consists of an input imageXX, a class attributeY∈𝒴=\{0,⋯,\|𝒴\|−1\}Y\\in\\mathcal\{Y\}=\\\{0,\\cdots,\|\\mathcal\{Y\}\|\-1\\\}and a sensitive attributeA∈\{0,1\}A\\in\\\{0,1\\\},*e\.g\.*, sex\.

#### Metrics\.

We measure three metrics for CF, GF, and classification accuracy\. Firstly, we describe the metric for CF\. A classifier satisfies CF when the predictions for the original sample and its counterfactual \(CTF\) sample are the same for every samplexxand sensitive attributeaa,*i\.e\.*,P​\(Y^​=​y\|X​=​x,A​=​a\)=P​\(Y^A←a′​=​y\|X​=​x,A​=​a\)P\(\\widehat\{Y\}\\text\{\\,\{=\}\\,\}y\|X\\text\{\\,\{=\}\\,\}x,A\\text\{\\,\{=\}\\,\}a\)=P\(\\widehat\{Y\}\_\{A\\leftarrow a^\{\\prime\}\}\\text\{\\,\{=\}\\,\}y\|X\\text\{\\,\{=\}\\,\}x,A\\text\{\\,\{=\}\\,\}a\), whereY^A←a′\\widehat\{Y\}\_\{A\\leftarrow a^\{\\prime\}\}represents the prediction for a counterfactual sample intervened onAAwitha′a^\{\\prime\}\(*e\.g\.*, changing female to male\)\. We quantify the degree of violence with respect to CF using counterfactual disparity \(CD\):

Counterfactual Fairness:Counterfactual Disparity\(CD\)≜𝔼x,a​\[P​\(𝟙​\{Y^A←a′≠Y^\}\|x,a\)\]\.\\displaystyle\\text\{ \{Counterfactual Fairness:\} Counterfactual Disparity \{\(CD\)\}\}\\triangleq\\mathbb\{E\}\_\{x,a\}\\big\[P\\big\(\\mathbbm\{1\}\\\{\\widehat\{Y\}\_\{A\\leftarrow a^\{\\prime\}\}\\neq\\widehat\{Y\}\\\}\|x,a\\big\)\\big\]\.\(1\)
Secondly, we adopt equalized odds \(EO\) as our notion for GF\. If a predictorY^\\widehat\{Y\}and the sensitive attributeAAare conditionally independent given the true class attributeYY, the predictor satisfies EO; namely, EO holds whenP​\(Y^​=​y′\|A​=​0,Y​=​y\)=P​\(Y^​=​y′\|A​=​1,Y​=​y\)P\(\\widehat\{Y\}\\text\{\\,\{=\}\\,\}y^\{\\prime\}\|A\\text\{\\,\{=\}\\,\}0,Y\\text\{\\,\{=\}\\,\}y\)=P\(\\widehat\{Y\}\\text\{\\,\{=\}\\,\}y^\{\\prime\}\|A\\text\{\\,\{=\}\\,\}1,Y\\text\{\\,\{=\}\\,\}y\)\. From the definition, we can capture the degree of violence with respect to GF with the disparity of EO \(DEO\):

Disparity of EO\(DEO\)≜maxy,y′∈𝒴\|P\\displaystyle\\text\{Disparity of EO \{\(DEO\)\}\}\\triangleq\\max\_\{y,y^\{\\prime\}\\in\\mathcal\{Y\}\}\\big\|P\(Y^=y′\|A=0,Y=y\)\-P\(Y^=y′\|A=1,Y=y\)\|\.\\displaystyle\(\\widehat\{Y\}\\text\{\\,\{=\}\\,\}y^\{\\prime\}\|A\\text\{\\,\{=\}\\,\}0,Y\\text\{\\,\{=\}\\,\}y\)\\text\{\{\-\}\}P\(\\widehat\{Y\}\\text\{\\,\{=\}\\,\}y^\{\\prime\}\|A\\text\{\\,\{=\}\\,\}1,Y\\text\{\\,\{=\}\\,\}y\)\\big\|\.\(2\)
We note that we empirically compute CD and DEO, defined in[Equation˜1](https://arxiv.org/html/2607.06603#S3.E1)and \([2](https://arxiv.org/html/2607.06603#S3.E2)\), using our benchmark datasets and the original test datasets of CelebA and LFW, respectively\. Additionally,Pintoet al\.\[[33](https://arxiv.org/html/2607.06603#bib.bib183)\]propose several other metrics to evaluate CF, and accordingly, we conducted an additional evaluation based on these metrics, with results provided in[Section˜G\.5](https://arxiv.org/html/2607.06603#A7.SS5)\.

#### Baseline methods\.

We evaluate a model trained with the vanilla cross\-entropy loss \(denoted as “Scratch”\) and two CF\-aware training methods, Scratch\+aug and counterfactual pairing \(CP\)\. Scratch\+aug is a Scratch method using an augmented training dataset with counterfactual samples\[[7](https://arxiv.org/html/2607.06603#bib.bib176)\], and CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]adopts a regularization term that promotes pairs of original and its CTF sample to obtain the same prediction \(see[Equation˜4](https://arxiv.org/html/2607.06603#S5.E4)for the formal definition\)\. Note both methods need counterfactual samples for training, and hence, we use the samples generated via IP2P with the same prompts used in[Section˜2](https://arxiv.org/html/2607.06603#S2)without any filtering process to obtain results for them\. For a comprehensive study, we additionally evaluate two individual fairness\-aware methods, SenSeI\[[42](https://arxiv.org/html/2607.06603#bib.bib25)\]and LASSI\[[31](https://arxiv.org/html/2607.06603#bib.bib24)\], of which goals are analogous to CF in aiming to make a model robust to perturbation of the sensitive attribute\. More details are described in[Section˜D\.3](https://arxiv.org/html/2607.06603#A4.SS3)\.

#### Model selection\.

Due to the accuracy\-fairness trade\-off\[[6](https://arxiv.org/html/2607.06603#bib.bib119)\], appropriate model selection is important for fair evaluation\. We explore varying hyperparameters and select the best model that shows the lowest CD \([Equation˜1](https://arxiv.org/html/2607.06603#S3.E1)\) for the held\-out validation set while achieving a lower bound of the accuracy222Considering the accuracy degradation of fair\-training methods, we set the bound as 98% of the accuracy of Scratch,*i\.e\.*, if Scratch achieves 95\.0% accuracy, then we only consider models with more than 93\.1% accuracy\.\.

Table 2:CF does not always imply GF on image classification\. We report CD \([Equation˜1](https://arxiv.org/html/2607.06603#S3.E1)\) and DEO \([Equation˜2](https://arxiv.org/html/2607.06603#S3.E2)\) for measuring Counterfactual Fairness \(CF\) and Group Fairness \(GF\), respectively\. Accuracy and DEO are measured on the original test datasets \(CelebA and LFW\) and CD is evaluated on the newly constructed datasets, CelebA\-CF and LFW\-CF, described in[Section˜2](https://arxiv.org/html/2607.06603#S2)\. If a model shows an inferior metric value than Scratch, the number is highlighted inred\.

### 3\.2Performance comparison

[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)shows accuracy, CD, and DEO for Scratch and four baseline methods\. Note that we omit the result of LASSI on LFW because the number of samples in LFW is not enough to train the Glow model\[[21](https://arxiv.org/html/2607.06603#bib.bib35)\], which is the main component of LASSI\. From the table, CF\-aware and individual fairness\-aware methods are mostly effective in mitigating CD, when compared to Scratch\. However, it does not necessarily lead to improvements in DEO\. Especially, while CP significantly improves CD for both datasets, it exacerbates DEO compared to Scratch\. Namely, contrary to the previous studies\[[1](https://arxiv.org/html/2607.06603#bib.bib51),[35](https://arxiv.org/html/2607.06603#bib.bib9)\]showing that CF implies GF on tabular datasets, our observation shows that CF does not always imply GF on image datasets\. In the following section, we theoretically investigate why the previous observations may not hold on images\.

## 4Theoritical analysis on the relationship between CF and EO for images

![Refer to caption](https://arxiv.org/html/2607.06603v1/x2.png)Figure 2:Image SCM\.Blue,gray, andyellowcircles representlatent attributes,components of an imageanda whole image, respectively\. Directed edges indicate a causal relationship from the source to the target\. The blue region indicates that there can be any direction of edges between blue nodes\.### 4\.1Structural Causal Model \(SCM\) for images

Structural Causal Models \(SCMs\) are represented as directed acyclic graphs satisfying the conditions specified in\[[30](https://arxiv.org/html/2607.06603#bib.bib17)\]\. In these models, nodes and edges indicate variables and their causal relationships within the data\-generating process\. As studied in previous works\[[5](https://arxiv.org/html/2607.06603#bib.bib49),[22](https://arxiv.org/html/2607.06603#bib.bib50)\], the nodes of an SCM for image can be categorized into three parts\. As shown in[Figure˜2](https://arxiv.org/html/2607.06603#S4.F2), theblue,grayandyellownodes indicatelatent attributes,*e\.g\.*,YYorAA,components of the image influenced by these attributes,*e\.g\.*,XYX\_\{Y\}orXAX\_\{A\}, andthe whole imageXX, respectively\. Taking an SCM for facial images as an example, we can interpret these nodes as follows: latent attributes such ashair color or sex, facial components like thehair or an Adam’s applein a facial image, andan entire face\. Note that the blue region in the figure describes that various causal relationships among latent attributes can exist333We assume no edge or unblocked path fromAAtoYY; otherwise, all counterfactually fair models based on that SCM would produce random predictions with respect toXYX\_\{Y\}\.\. Furthermore, although an image SCM may contain additional latent attributes, we simplify our focus to only include the class and sensitive attribute,YYandAA, and a third\-party attribute,GG, which may correlate with the sensitive attributeAA\.

### 4\.2Theoretical analysis

According to the Markov assumption of SCM\[[30](https://arxiv.org/html/2607.06603#bib.bib17)\], if there are no unblocked paths between two variables in an SCM \(*i\.e\.*, they are d\-separated\), the variables are statistically independent\. Utilizing this property,Anthis and Veitch \[[1](https://arxiv.org/html/2607.06603#bib.bib51)\]demonstrated that CF implies several GF notions, including Equalized Odds \(EO\)\[[10](https://arxiv.org/html/2607.06603#bib.bib127)\], under the specific condition on SCMs such as nobackdoorpath from the sensitive attributeAAto the imageXXexists \(Theorem 2 of\[[1](https://arxiv.org/html/2607.06603#bib.bib51)\]\)\. Moreover, the authors empirically show that these conditions would hold on some tabular datasets\.

However, we argue that these conditions would not hold for image datasets due to a fundamental difference in what sensitive attributes represent in an image\. Specifically, tabular datasets typically consist of recorded information by subjects, where sensitive attributes such as sex or race usually represent immutable genetic information; hence they are not caused by other attributes and cause all attributes correlated with the sensitive attributes\. In contrast, sensitive attributes in image datasets indicate visual characteristics that can change and be influenced by some other attributes, such as the attributeGG\. For example, in a facial image dataset, attributes like hair length or accessories might be highly correlated with, but not caused by, secondary sex characteristics such as beard\. Namely, a backdoor path from the sensitive attributeXXthrough the attributeGGcould exist, thereby breaking the connection between CF and GF discovered in previous studies\.

Our theoretical result specifies the relationship between CF and GF \(especially for EO\) withGG:

###### Theorem 4\.1\.

Assume a latent attributeGGin[Figure˜2](https://arxiv.org/html/2607.06603#S4.F2)is a non\-descendant variable ofAAand connected toAAthrough an unblocked path\. Then, the following inequality holds for a counterfactually fair classifier𝛉\\bm\{\\theta\}and any pairs ofyyandy′y^\{\\prime\}:

\|P\(Y^=y′\|A=0,Y=y\)−P\(Y^=y′\|A=1,Y=y\)\\displaystyle\\big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|A=0,Y=y\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|A=1,Y=y\)\|\\displaystyle\\big\|≤∑XYP​\(XY\|Y=y\)\\displaystyle\\leq\\sum\_\{X\_\{Y\}\}P\(X\_\{Y\}\|Y=y\)maxXG,XG′⁡d𝜽,XY​\(XG,XG′\),\\displaystyle\\max\_\{X\_\{G\},X\_\{G\}^\{\\prime\}\}d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X\_\{G\}^\{\\prime\}\),\(3\)
in whichd𝛉,XY\(XG,XG′\)=\|P\(Y^=y′\|XY,XG\)−P\(Y^=y′\|XY,XG′\)\|d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X\_\{G\}^\{\\prime\}\)=\\big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}^\{\\prime\}\)\\big\|andY^\\hat\{Y\}is the prediction of the model𝛉\\bm\{\\theta\}\. The equality holds whend𝛉,XY​=​0d\_\{\\bm\{\\theta\},X\_\{Y\}\}\\text\{\\,\{=\}\\,\}0always regardless ofXYX\_\{Y\}\.

The proof of the theorem is in[Appendix˜A](https://arxiv.org/html/2607.06603#A1)\. Note that when we take the maximum over\(y,y′\)\(y,y^\{\\prime\}\)on both sides of the inequality in[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1), the left\-hand side of the inequality becomes identical to DEO \([Equation˜2](https://arxiv.org/html/2607.06603#S3.E2)\)\. Therefore, the theorem implies that DEO is upper bounded by the maximum ofd𝜽,XY​\(XG,XG′\)d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X\_\{G\}^\{\\prime\}\)\(in which the maximum is overXGX\_\{G\},XG′X\_\{G\}^\{\\prime\},yy,y′y^\{\\prime\}\), which measures the sensitivity of the model with respect toGG\. In other words, the theorem shows that when a counterfactually fair model is sensitive toXGX\_\{G\}\(*i\.e\.*, whenmax⁡d𝜽,XY​\(XG,XG′\)\\max d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X^\{\\prime\}\_\{G\}\)is large\), the model may result in having high DEO in the worst\-case\.

[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)elucidates why CF\-aware methods in[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)often fail to mitigate DEO despite significant improvements in CD\. Namely, if the attributeGGassumed in[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)exists on CelebA and LFW, DEO for the classifiers trained by CF\-aware methods can worsen depending on their robustness toGG\. This will be empirically demonstrated using “hair length” asGGin[Section˜5\.2](https://arxiv.org/html/2607.06603#S5.SS2), together with the results using a controllable synthetic dataset\. Furthermore,[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)suggests that we can re\-establish the relationship between two notions by making counterfactually fair classifiers non\-sensitive toGG\. In the following section, we introduce a method to promote a classifier not to depend onGGwhile achieving CF\.

## 5Empirical analyses on the effect ofGGto CF and GF

### 5\.1Counterfactual Knowledge Distillation \(CKD\)

Motivated by[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1), we propose a baseline fair\-training method to achieve both CF and GF\. Conceptually, if we can reduce the dependency between the latent attributeGGdescribed in Theorem[4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)and the prediction of a CF\-aware trained model, we can expect that the model will achieve CF and GF simultaneously\. Therefore, we improve the CF\-aware method, CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]\(best\-performing in[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)\), such that the dependency to the attributeGGis reduced\. We first describe the CP regularization \(which is used along with the cross\-entropy loss\) for given counterfactual samples𝒟′=\{xi,A←ai′\}i=1N\\mathcal\{D^\{\\prime\}\}=\\\{x\_\{i,A\\leftarrow a\_\{i\}^\{\\prime\}\}\\\}\_\{i=1\}^\{N\}corresponding to the original training dataset𝒟\\mathcal\{D\}:

ℒCP\\displaystyle\\mathcal\{L\}\_\{\\text\{CP\}\}\(θ,𝒟∪𝒟′\):=1N​∑i=1N∥f​\(𝜽,xi\)−f​\(𝜽,xi,A←ai′\)∥22,\\displaystyle\(\\theta,\\mathcal\{D\}\\cup\\mathcal\{D^\{\\prime\}\}\):=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\lVert f\(\\bm\{\\theta\},x\_\{i\}\)\-f\(\\bm\{\\theta\},x\_\{i,A\\leftarrow a\_\{i\}^\{\\prime\}\}\)\\rVert\_\{2\}^\{2\},\(4\)
in whichf​\(𝜽,x\)f\(\\bm\{\\theta\},x\)is a representation vector of inputxxproduced by a classifier𝜽\\bm\{\\theta\}, such as logit or feature vector\. Note that the imagesxxandxA←a′x\_\{A\\leftarrow a^\{\\prime\}\}differ only in their components corresponding to the sensitive attributeAAand the attributes caused by the sensitive attributeAA\. Hence, although the CP regularization works well for achieving CF, it does not ensure the model does not rely on the attributeGG, potentially leading to worse DEO as argued in the previous section\.

Recent studies\[[16](https://arxiv.org/html/2607.06603#bib.bib86),[38](https://arxiv.org/html/2607.06603#bib.bib32),[45](https://arxiv.org/html/2607.06603#bib.bib52)\]have shown that the robustness of a teacher model can be transferred into a student model through knowledge distillation \(KD\)\[[12](https://arxiv.org/html/2607.06603#bib.bib137)\]\. To that end, we first assume a teacher model that is robust to the attributeGGis available\. Then, our idea is to apply both KD and CP regularization to train our student model, which leads to a simple yet effective approach, dubbed as Counterfactual Knowledge Distillation \(CKD\)\. Specifically, CKD employs averaged representation vectors of original and counterfactual samples extracted by the teacher model𝜽T\\bm\{\\theta\}^\{T\}as target vectors\. Then, representation vectors of both samples from the student model𝜽\\bm\{\\theta\}are enforced to follow the target vectors\. Namely, the distillation term of CKD is defined as follows:

ℒ\\displaystyle\\mathcal\{L\}\(𝜽,𝒟∪𝒟′\)C​K​D:=12​N∑iN\(∥f\(𝜽,xi\)−fiT∥22\+∥f\(𝜽,xi,A←ai′\)−fiT∥22\),\{\}\_\{CKD\}\(\\bm\{\\theta\},\\mathcal\{D\}\\cup\\mathcal\{D\}^\{\\prime\}\):=\\frac\{1\}\{2N\}\\sum\_\{i\}^\{N\}\\bigg\(\\lVert f\(\\bm\{\\theta\},x\_\{i\}\)\-f^\{T\}\_\{i\}\\rVert\_\{2\}^\{2\}\+\\lVert f\(\\bm\{\\theta\},x\_\{i,A\\leftarrow a\_\{i\}^\{\\prime\}\}\)\-f^\{T\}\_\{i\}\\rVert\_\{2\}^\{2\}\\bigg\),in whichfiT=12​\(f​\(𝜽T,xi\)\+f​\(𝜽T,xi,A←ai′\)\)is the target vector for thei\-th pair\.\\displaystyle\\text\{in which\}\\ \\ f^\{T\}\_\{i\}=\\frac\{1\}\{2\}\\big\(f\(\\bm\{\\theta\}^\{T\},x\_\{i\}\)\+f\(\\bm\{\\theta\}^\{T\},x\_\{i,A\\leftarrow a\_\{i\}^\{\\prime\}\}\)\\big\)\\ \\ \\text\{is the target vector for the $i$\-th pair\.\}\(5\)Note that our distillation terms have both effects of KD and CP by promoting both representations of original and counterfactual samples to be aligned with the target vectorsfiTf\_\{i\}^\{T\}produced by the teacher model\. Therefore, based on[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1), we can deduce that the CKD regularization encourages the model to achieve both CF \(by the CP effect\) and EO\-based GF \(by the KD effect that distills the robustness of the teacher with respect to the attributeGG\)\. In addition, we optionally incorporate CP regularization into our objective to further mitigate CD\. The final objective of our method \(which we again dub as CKD for brevity\) is as follows:

min𝜽\\displaystyle\\underset\{\\bm\{\\theta\}\}\{\\min\}\\ℒCE​\(𝜽,𝒟\)\+μ​ℒCKD​\(𝜽,𝒟∪𝒟′\)\+λ​ℒCP​\(𝜽,𝒟∪𝒟′\),\\displaystyle\\mathcal\{L\}\_\{\\text\{CE\}\}\(\\bm\{\\theta\},\\mathcal\{D\}\)\+\\mu\\mathcal\{L\}\_\{\\text\{CKD\}\}\(\\bm\{\\theta\},\\mathcal\{D\}\\cup\\mathcal\{D\}^\{\\prime\}\)\+\\lambda\\mathcal\{L\}\_\{\\text\{CP\}\}\(\\bm\{\\theta\},\\mathcal\{D\}\\cup\\mathcal\{D\}^\{\\prime\}\),\(6\)in whichμ\\muandλ\\lambdaare controllable hyperparameters for the CKD and CP regularization, respectively\.

While we assumed above the availability of a teacher model that is robust to the attributeGG, obtaining such a model could be challenging in practice\. Empirically, we observe that vanilla\-trained models \(referred to as “Scratch” models\) less depend on the attributeGGthan CP\-trained ones \(see[Figure˜3](https://arxiv.org/html/2607.06603#S5.F3)and[Table˜3](https://arxiv.org/html/2607.06603#S5.T3)for more details\)\. We presume that this is because the attributesAAandGGbehave as “shortcut” features for classifying the class attributeYY,*i\.e\.*, they are easy\-to\-learn discriminatory features\. As observed byScimecaet al\.\[[37](https://arxiv.org/html/2607.06603#bib.bib182)\], making a model blind to a certain shortcut feature causes it to rely more heavily on the other shortcut features\. In our case, CP\-trained models are trained to be invariant to the sensitive attributeAA, resulting in a greater dependence on the attributeGGcompared to the Scratch models\. Thus, unless otherwise specified, we will assume the vanilla\-trained model is relatively robust to the attributeGGsince it would mostly rely on the sensitive attributeAA, hence, we use it as the teacher model\.

### 5\.2Impact of robustness toGGon CF and GF

We empirically validate our theoretical result and CKD on both a newly introduced synthetic dataset \(CIFAR\-10B\) and a real\-world dataset \(CelebA\) by analyzing CF, GF, and the robustness with respect to the attributeGGdescribed in Theorem[4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)\. We thus introduce a new metric for the robustness to the attributeGG, the rate of flipped predictions \(RFP\) :

RFP≜𝔼x,x′​\[P​\(𝟙​\{Y^≠Y^′\}\|x,x′\)\]\.\\displaystyle\\text\{RFP\}\\triangleq\\mathbb\{E\}\_\{x,x^\{\\prime\}\}\\big\[P\\big\(\\mathbbm\{1\}\\\{\\widehat\{Y\}\\neq\\widehat\{Y\}^\{\\prime\}\\\}\|x,x^\{\\prime\}\\big\)\\big\]\.\(7\)in whichxxis an original image,x′x^\{\\prime\}is its corresponding image with the attributeGGflipped\.Y^\\widehat\{Y\}andY^′\\widehat\{Y\}^\{\\prime\}refer to the predicted label by the trained model givenxxandx′x^\{\\prime\}, respectively\. RFP quantifies the amount of flipped predictions when the attributeGGis altered\. For example, if a model shows the same prediction after changing the attributeGG, its RFP becomes 0%\.

#### CIFAR\-10B, a controllable synthetic dataset\.

We construct the CIFAR\-10B dataset, where we can perfectly control the degree of bias with respect to the attributeGGwhile the target label is biased towards the sensitive attributeAA\. We make binary class labels from the 10 classes of CIFAR\-10 \(0\-4 and 5\-9 classes\)\. We set the attributesAAandGGin[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)with the presence of Gaussian and Contrast noise, respectively\. We also set a fixed ratio of 0\.8 and a controllable ratioα\\alpha, which represent skewnesses among \(YY,AA\) and \(AA,GG\), respectively; the former ratio is the spurious correlation betweenYYandAA, and the latter one is the correlation betweenAAandGG\. We then construct the CIFAR\-10B dataset by randomly injecting Gaussian or Contrast noise to each CIFAR\-10 image at given ratios, as illustrated in[Figure˜E\.2](https://arxiv.org/html/2607.06603#A5.F2)\. Unless otherwise noted, we setα\\alphaas 0\.8\.

We train models with Scratch, CP, and CKD on CIFAR\-10B by adjustingα\\alphafrom 0\.5 \(*i\.e\.*,AAandGGare decorrelated\) to 0\.9 at intervals of 0\.1\.[Figure˜3](https://arxiv.org/html/2607.06603#S5.F3)shows CD, DEO, and RFP metric values for each method\. The figures indicate that while CP and our CKD consistently achieve CF, CP fails to meet GF asα\\alphaincreases, potentially due to higher RFP\. Furthermore, RFP of Scratch is lower than that of CP whenα\\alphais greater than 0\.7\. This empirically justifies the use of vanilla\-trained models as teacher models robust toGG\. By using these teacher models, CKD significantly improves DEO regardless of the value ofα\\alphaby maintaining the robustness toGG,*i\.e\.*, low RFP, supporting the result of[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)\.

![Refer to caption](https://arxiv.org/html/2607.06603v1/x3.png)Figure 3:Impact of the correlation ofGGandAA\.α\\alphaindicates howAAandGGare correlated on CIFAR\-10B\.Table 3:Impact ofGGon CelebA\.We assume “hair length” asGGand manipulate the hair length of test images\. CD, DEO, and RFP are measured on CelebA\-CF, CelebA, and hair\-edited CelebA, respectively\.
#### Impact ofGGmanipulation on CelebA\.

We assume “hair length” asGGfor facial image datasets,*e\.g\.*, CelebA because the hair lengthGGcan be highly correlated with, but not caused by, the sexAA\. To compute RFP for the hair length attribute, we manipulate the hair length of CelebA test images using SDEdit\[[28](https://arxiv.org/html/2607.06603#bib.bib29)\]\. More details and generated examples can be found in[Section˜E\.1](https://arxiv.org/html/2607.06603#A5.SS1)\. Using the hair length\-edited images, we report RFP in[Table˜3](https://arxiv.org/html/2607.06603#S5.T3), together with DEO and CD\. The results share the same trend as the CIFAR\-10B results,*i\.e\.*, CP shows worse DEO and RFP than Scratch but better CD, whereas CKD shows the best DEO and RFP, despite a slight increase of CD\.

### 5\.3Impact of the robustness toGGof the teacher model on CKD

Table 4:Impact of robustness toGGof the teacher model\.𝜽CPT\\bm\{\\theta\}^\{T\}\_\{\\text\{CP\}\},𝜽CKDT\\bm\{\\theta\}^\{T\}\_\{\\text\{CKD\}\}, and𝜽ScratchT\\bm\{\\theta\}^\{T\}\_\{\\text\{Scratch\}\}are CP, CKD, and Scratch teacher model\.𝜽De\-biasedT\\bm\{\\theta\}^\{T\}\_\{\\text\{De\-biased\}\}is a Scratch model trained on a perfectly de\-biased training dataset \(α=0\.5\\alpha=0\.5\)\. RFPTdenotes how a teacher is biased towardsGG\. CD, DEO, RFP are metrics for evaluating CF, GF, and bias towardsGG, respectively\. Results are measured on CIFAR\-10B withα=0\.8\\alpha=0\.8Our CKD requires a robust teacher model with respect to the attributeGGto distill the robustness to the target model\. To analyze the impact of the robustness of the teacher model, we compare various teacher models with different dependencies on the attributeGGusing CIFAR\-10B\. We consider four teacher models, ordered by robustness to the attributeGG: CP \(𝜽CPT\\bm\{\\theta\}^\{T\}\_\{\\text\{CP\}\}\), Scratch \(𝜽ScratchT\\bm\{\\theta\}^\{T\}\_\{\\text\{Scratch\}\}\), and CKD model with a Scratch teacher \(𝜽CKDT\\bm\{\\theta\}^\{T\}\_\{\\text\{CKD\}\}\), and a de\-biased model trained on CIFAR\-10B balanced forGG,*i\.e\.*,α=0\.5\\alpha=0\.5, \(𝜽De\-biasedT\\bm\{\\theta\}^\{T\}\_\{\\text\{De\-biased\}\}\)\. Using these teacher models, we report DEO, CD, and RFP of CKDs on the CIFAR\-10B dataset in[Table˜4](https://arxiv.org/html/2607.06603#S5.T4)\. We observe that the degree of robustness to the attributeGGof the teacher model \(*i\.e\.*, RFPT\) highly correlates to DEO\. It is because as the teacher model becomes more robust toGG, RFP of the target model gets lower, finally leading to a lower DEO while maintaining fair CD\. Namely, these results support our theoretical result again\.

## 6Full comparisons of fair\-training methods on image classification

Finally, we evaluate the existing fair\-training methods focusing on group fairness \(GF\) and counterfactual fairness \(CF\) on CelebA and LFW, together with CIFAR\-10B for image classification tasks\. We emphasize that only CelebA\-CF and LFW\-CF have counterfactual images of the real\-world images; hence, we measure a CF metric,*i\.e\.*, Counterfactual Disparity \(CD\) \([Equation˜1](https://arxiv.org/html/2607.06603#S3.E1)\), using our datasets\. Along with the CF\-aware methods, such as CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]and CKD, we report the GF\-aware methods including SS\[[14](https://arxiv.org/html/2607.06603#bib.bib58)\], RW\[[17](https://arxiv.org/html/2607.06603#bib.bib82)\], COV\[[43](https://arxiv.org/html/2607.06603#bib.bib83)\], MFD\[[16](https://arxiv.org/html/2607.06603#bib.bib86)\], and LBC\[[15](https://arxiv.org/html/2607.06603#bib.bib89)\]\. In addition, we report the naive combinations of GF\-aware and CF\-aware methods,*e\.g\.*, training GF\-aware method with the augmented training dataset with counterfactual images generated by IP2P\[[2](https://arxiv.org/html/2607.06603#bib.bib26)\]\(denoted as “\+aug”\) and combinations of the GF\-aware methods and the CP regularization \([Equation˜4](https://arxiv.org/html/2607.06603#S5.E4)\) \(denoted as “\+CP”\)\. The hyperparameters for all methods besides the GF\-aware methods are selected using the same protocol in[Section˜3](https://arxiv.org/html/2607.06603#S3), and ones for the GF\-aware methods are chosen based on DEO using the same lower bound of the accuracy\. Implementation details are provided in[Section˜D\.3](https://arxiv.org/html/2607.06603#A4.SS3)\.

Table 5:Evaluation of GF and CF of fair\-training for image classification\. The details are the same as[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)\. “Scratch” denotes a model trained without considering the notion of fairness through the vanilla cross\-entropy loss\. “\+aug” denotes counterfactual \(CTF\) image augmentation described in[Section˜3](https://arxiv.org/html/2607.06603#S3)\. If a model performs worse than the Scrath model on CD/DEO, we highlight the numbers inred\. The best performance is highlighted inorange, and the second\-best performance is highlighted ingrey\.[Table˜H\.1](https://arxiv.org/html/2607.06603#A8.T1)shows the holistic evaluation of CF and GF for all the methods mentioned above on the three image classification tasks\. The table shows four important observations\. First, although CP \(a CF\-aware method\) mostly performs the best on CD, it even shows worse DEO than Scratch\. We theoretically and empirically discussed the reason in[Section˜4](https://arxiv.org/html/2607.06603#S4)and[5](https://arxiv.org/html/2607.06603#S5)\. Second, the GF\-aware methods are effective in improving DEO but have a minimal impact on CD\. This suggests that the faithfulness assumption for SCM may not hold, which will be discussed in more details in[Appendix˜F](https://arxiv.org/html/2607.06603#A6)\. Third, the naive combinations of GF\-aware and CF\-aware methods exhibit much better CD than using the GF\-aware methods alone\. Additionally, DEOs achieved by the naively combined methods tend to be improved since their training datasets are balanced over the sensitive attributes by incorporating generated samples into the original training datasets\. Lastly, we found that CKD shows the best DEO for every evaluation dataset\. It shows that if we can train a CF\-aware model by reducing the dependency onGG, we can achieve both CF and GF even on the image classification task\. We additionally conduct an ablation study on CKD by removing CP \(*i\.e\.*, CKD \(λ=0\\lambda=0\)\) from[Equation˜6](https://arxiv.org/html/2607.06603#S5.E6)\. We observe that this ablated version achieves suboptimal performances than CKD\. This suggests that adding the CP regularization term to the CKD objective function can be helpful to improve both CD and DEO\.

## 7Concluding remarks

This paper offers carefully crafted benchmark datasets for evaluating the counterfactual fairness \(CF\) of image classification methods\. Since obtaining true counterfactual images is impossible in practice, we employ a high\-quality image editing technique to generate counterfactual images of the given images\. We construct two facial image benchmarks, CelebA\-CF and LFW\-CF, by carefully filtering out and verifying the generated counterfactual images by human annotators\. Our datasets relax the constraints of the impossibility of evaluating CF in image classification\. Using our datasets, we also provide theoretical and empirical results showing that CF may not imply GF, contradictory to the studies conducted on tabular datasets\. We elucidate this phenomenon by the presence of the third\-party attribute highly correlated with, but not caused by, the sensitive attribute\. From this finding, we propose a simple baseline method, CKD, to achieve CF and GF simultaneously\. Our extensive experimental results on both GF and CF metrics show that when reducing the reliance on the attribute \(*e\.g\.*, by using CKD\), improving the CF metric leads to a significant improvement in the GF metric\. By providing our benchmarks and various analyses, we believe that our findings bridge CF and GF in image classification, contributing to the development of fair and robust image recognition systems\.

## Acknowledgments

This work was supported in part by the National Research Foundation of Korea \(NRF\) grant \[No\.2021R1A2C2007884\] and by Institute of Information & communications Technology Planning & Evaluation \(IITP\) grants \[RS\-2021\-II211343, RS\-2021\-II212068, RS\-2022\-II220113, RS\-2022\-II220959\] funded by the Korean government \(MSIT\)\. It was also supported by AOARD Grant No\. FA2386\-23\-1\-4079, SNU\-Naver Hyperscale AI Center, and Hyundai Motor Chung Mong\-Koo Foundation\.

## References

- \[1\]J\. R\. Anthis and V\. Veitch\(2023\)Causal context connects counterfactual fairness to robust prediction and group fairness\.InAdv\. Neural Inform\. Process\. Syst\. \(NeurIPS\),Cited by:[Appendix F](https://arxiv.org/html/2607.06603#A6.p1.1),[§1](https://arxiv.org/html/2607.06603#S1.p2.1),[§1](https://arxiv.org/html/2607.06603#S1.p5.6),[§3\.2](https://arxiv.org/html/2607.06603#S3.SS2.p1.1),[§4\.2](https://arxiv.org/html/2607.06603#S4.SS2.p1.2)\.
- \[2\]T\. Brooks, A\. Holynski, and A\. A\. Efros\(2023\)Instructpix2pix: learning to follow image editing instructions\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\. \(CVPR\),pp\. 18392–18402\.Cited by:[3rd item](https://arxiv.org/html/2607.06603#A3.I1.i3.p1.1),[§H\.5](https://arxiv.org/html/2607.06603#A8.SS5.p4.1),[item 4a](https://arxiv.org/html/2607.06603#Ax1.I2.i4.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.06603#S1.p4.1),[§2](https://arxiv.org/html/2607.06603#S2.p2.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[3\]J\. Buolamwini and T\. Gebru\(2018\)Gender shades: intersectional accuracy disparities in commercial gender classification\.InConf\. Fairness, Accountability and Transparency \(FAccT\),pp\. 77–91\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p1.1)\.
- \[4\]M\. D’Incà, C\. Tzelepis, I\. Patras, and N\. Sebe\(2024\)Improving fairness using vision\-language driven image augmentation\.InIEEE/CVF Winter Conf\. App\. Comput\. Vis\. \(WACV\),pp\. 4695–4704\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[5\]S\. Dash, V\. N\. Balasubramanian, and A\. Sharma\(2022\)Evaluating and mitigating bias in image classifiers: a causal perspective using counterfactuals\.InIEEE/CVF Winter Conf\. App\. Comput\. Vis\. \(WACV\),pp\. 915–924\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1),[§4\.1](https://arxiv.org/html/2607.06603#S4.SS1.p1.9)\.
- \[6\]S\. Dutta, D\. Wei, H\. Yueksel, P\. Chen, S\. Liu, and K\. Varshney\(2020\)Is there a trade\-off between fairness and accuracy? A perspective using mismatched hypothesis testing\.InInt\. Conf\. Mach\. Learn\. \(ICML\),pp\. 2803–2813\.Cited by:[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px3.p1.1)\.
- \[7\]S\. Garg, V\. Perot, N\. Limtiaco, A\. Taly, E\. H\. Chi, and A\. Beutel\(2019\)Counterfactual fairness in text classification through robustness\.InAAAI/ACM Conf\. AI, Ethics, and Society \(AIES\),pp\. 219–226\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p1.1),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.10.10.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.18.18.7),[§1](https://arxiv.org/html/2607.06603#S1.p5.6),[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.06603#S3.T2.6.9.1.1)\.
- \[8\]M\. Goldblum, L\. Fowl, S\. Feizi, and T\. Goldstein\(2020\)Adversarially robust distillation\.InProc\. of the AAAI Conf\. Artificial Intelligence \(AAAI\),Vol\.34,pp\. 3996–4003\.Cited by:[§G\.3](https://arxiv.org/html/2607.06603#A7.SS3.p4.3)\.
- \[9\]I\. Goodfellow, J\. Pouget\-Abadie, M\. Mirza, B\. Xu, D\. Warde\-Farley, S\. Ozair, A\. Courville, and Y\. Bengio\(2014\)Generative adversarial nets\.Adv\. Neural Inform\. Process\. Syst\. \(NeurIPS\)27\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[10\]M\. Hardt, E\. Price, and N\. Srebro\(2016\)Equality of opportunity in supervised learning\.InAdv\. Neural Inform\. Process\. Syst\. \(NeurIPS\),Vol\.29\.Cited by:[§4\.2](https://arxiv.org/html/2607.06603#S4.SS2.p1.2)\.
- \[11\]E\. Harlan and O\. Schnuck\(2021\-02\-162021\-02\-16\)Objective or biased – the questionable use of artificial intelligence in job applications\.bayerischer rundfunk\.External Links:[Link](https://interaktiv.br.de/ki-bewerbung/en/)Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p1.1)\.
- \[12\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§G\.3](https://arxiv.org/html/2607.06603#A7.SS3.p1.2),[Table G\.5](https://arxiv.org/html/2607.06603#A7.T5),[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p3.3)\.
- \[13\]G\. B\. Huang, V\. Jain, and E\. Learned\-Miller\(2007\)Unsupervised joint alignment of complex images\.InInt\. Conf\. Comput\. Vis\. \(ICCV\),Cited by:[2nd item](https://arxiv.org/html/2607.06603#A3.I1.i2.p1.1),[§D\.1](https://arxiv.org/html/2607.06603#A4.SS1.p1.1),[§H\.2](https://arxiv.org/html/2607.06603#A8.SS2.p1.1),[item 4a](https://arxiv.org/html/2607.06603#Ax1.I2.i4.I1.i1.p1.1),[§2](https://arxiv.org/html/2607.06603#S2.p3.1)\.
- \[14\]B\. Y\. Idrissi, M\. Arjovsky, M\. Pezeshki, and D\. Lopez\-Paz\(2022\)Simple data balancing achieves competitive worst\-group\-accuracy\.InConf\. Causal Learning and Reasoning \(CLeaR\),pp\. 336–351\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p2.1),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.19.19.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.39.39.7),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.13.2.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[15\]H\. Jiang and O\. Nachum\(2020\)Identifying and correcting label bias in machine learning\.InInt\. Conf\. Artificial Intelligence and Statistics \(AISTATS\),pp\. 702–712\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p2.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.12.12.3),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.31.31.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.63.63.7),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.17.6.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[16\]S\. Jung, D\. Lee, T\. Park, and T\. Moon\(2021\)Fair feature distillation for visual recognition\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\. \(CVPR\),pp\. 12115–12124\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p2.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.10.10.3),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.28.28.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.57.57.7),[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p3.3),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.16.5.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[17\]F\. Kamiran and T\. Calders\(2012\)Data preprocessing techniques for classification without discrimination\.Knowledge and Information Systems \(KAIS\)33\(1\),pp\. 1–33\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p2.1),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.22.22.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.45.45.7),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.14.3.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[18\]R\. Kennaway\(2020\)When causation does not imply correlation: robust violations of the faithfulness axiom\.InThe Interdisciplinary Handbook of Perceptual Control Theory,pp\. 49–72\.Cited by:[Appendix F](https://arxiv.org/html/2607.06603#A6.p1.1)\.
- \[19\]H\. Kim, S\. Shin, J\. Jang, K\. Song, W\. Joo, W\. Kang, and I\. Moon\(2021\)Counterfactual fairness with disentangled causal effect variational autoencoder\.InProc\. of the AAAI Conf\. Artificial Intelligence \(AAAI\),Vol\.35,pp\. 8128–8136\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1),[§2](https://arxiv.org/html/2607.06603#S2.p2.1)\.
- \[20\]D\. P\. Kingma and M\. Welling\(2013\)Auto\-encoding variational bayes\.arXiv preprint arXiv:1312\.6114\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[21\]D\. P\. Kingma and P\. Dhariwal\(2018\)Glow: generative flow with invertible 1x1 convolutions\.Adv\. Neural Inform\. Process\. Syst\. \(NeurIPS\)31\.Cited by:[§3\.2](https://arxiv.org/html/2607.06603#S3.SS2.p1.1)\.
- \[22\]M\. Kocaoglu, C\. Snyder, A\. G\. Dimakis, and S\. Vishwanath\(2017\)Causalgan: learning causal implicit generative models with adversarial training\.arXiv preprint arXiv:1709\.02023\.Cited by:[§2](https://arxiv.org/html/2607.06603#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.06603#S4.SS1.p1.9)\.
- \[23\]M\. J\. Kusner, J\. R\. Loftus, C\. Russell, and R\. Silva\(2017\)Counterfactual fairness\.InAdv\. Neural Inform\. Process\. Syst\. \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p2.1)\.
- \[24\]H\. Liang, P\. Perona, and G\. Balakrishnan\(2023\)Benchmarking algorithmic bias in face recognition: an experimental approach using synthetic faces and human evaluation\.InInt\. Conf\. Comput\. Vis\. \(ICCV\),pp\. 4977–4987\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[25\]Z\. Liu, P\. Luo, X\. Wang, and X\. Tang\(2015\)Deep learning face attributes in the wild\.InInt\. Conf\. Comput\. Vis\. \(ICCV\),pp\. 3730–3738\.Cited by:[1st item](https://arxiv.org/html/2607.06603#A3.I1.i1.p1.1),[§D\.1](https://arxiv.org/html/2607.06603#A4.SS1.p1.1),[§H\.2](https://arxiv.org/html/2607.06603#A8.SS2.p1.1),[item 4a](https://arxiv.org/html/2607.06603#Ax1.I2.i4.I1.i1.p1.1),[§2](https://arxiv.org/html/2607.06603#S2.p3.1)\.
- \[26\]C\. Louizos, U\. Shalit, J\. M\. Mooij, D\. Sontag, R\. Zemel, and M\. Welling\(2017\)Causal effect inference with deep latent\-variable models\.Adv\. Neural Inform\. Process\. Syst\. \(NeurIPS\)30\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[27\]A\. S\. Luccioni, C\. Akiki, M\. Mitchell, and Y\. Jernite\(2023\)Stable bias: analyzing societal representations in diffusion models\.arXiv preprint arXiv:2303\.11408\.Cited by:[§H\.5](https://arxiv.org/html/2607.06603#A8.SS5.p4.1)\.
- \[28\]C\. Meng, Y\. He, Y\. Song, J\. Song, J\. Wu, J\. Zhu, and S\. Ermon\(2021\)Sdedit: guided image synthesis and editing with stochastic differential equations\.arXiv preprint arXiv:2108\.01073\.Cited by:[§E\.1](https://arxiv.org/html/2607.06603#A5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2607.06603#S5.SS2.SSS0.Px2.p1.3)\.
- \[29\]L\. S\. Nguyen and D\. Gatica\-Perez\(2016\)Hirability in the wild: analysis of online conversational video resumes\.IEEE Trans\. Multimedia18\(7\),pp\. 1422–1437\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p1.1)\.
- \[30\]J\. Pearlet al\.\(2000\)Models, reasoning and inference\.Cambridge, UK: CambridgeUniversityPress19\(2\),pp\. 3\.Cited by:[§4\.1](https://arxiv.org/html/2607.06603#S4.SS1.p1.9),[§4\.2](https://arxiv.org/html/2607.06603#S4.SS2.p1.2)\.
- \[31\]M\. Peychev, A\. Ruoss, M\. Balunović, M\. Baader, and M\. Vechev\(2022\)Latent space smoothing for individually fair representations\.InEur\. Conf\. Comput\. Vis\. \(ECCV\),pp\. 535–554\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p1.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.6.6.3),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.33.33.4),[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.06603#S3.T2.6.12.4.1)\.
- \[32\]S\. R\. Pfohl, T\. Duan, D\. Y\. Ding, and N\. H\. Shah\(2019\)Counterfactual reasoning for fair clinical risk prediction\.InMachine Learning for Healthcare Conference \(MLHC\),pp\. 325–358\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1)\.
- \[33\]M\. Pinto, A\. V\. Carreiro, P\. Madeira, A\. Lopez, and H\. Gamboa\(2024\)The matrix reloaded: towards counterfactual group fairness in machine learning\.Journal of Data\-centric Machine Learning Research \(DMLR\)\.Cited by:[§G\.5](https://arxiv.org/html/2607.06603#A7.SS5),[§G\.5](https://arxiv.org/html/2607.06603#A7.SS5.p1.4),[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px1.p5.1)\.
- \[34\]V\. V\. Ramaswamy, S\. S\. Kim, and O\. Russakovsky\(2021\)Fair attribute classification through latent space de\-biasing\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\. \(CVPR\),pp\. 9301–9310\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1),[§2](https://arxiv.org/html/2607.06603#S2.p2.1)\.
- \[35\]L\. Rosenblatt and R\. T\. Witter\(2023\)Counterfactual fairness is basically demographic parity\.InProc\. of the AAAI Conf\. Artificial Intelligence \(AAAI\),Vol\.37,pp\. 14461–14469\.Cited by:[Appendix F](https://arxiv.org/html/2607.06603#A6.p1.1),[§1](https://arxiv.org/html/2607.06603#S1.p2.1),[§1](https://arxiv.org/html/2607.06603#S1.p5.6),[§3\.2](https://arxiv.org/html/2607.06603#S3.SS2.p1.1)\.
- \[36\]C\. Russell, M\. J\. Kusner, J\. Loftus, and R\. Silva\(2017\)When worlds collide: integrating different counterfactual assumptions in fairness\.Adv\. Neural Inform\. Process\. Syst\. \(NeurIPS\)30\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p1.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.2.2.3),[§G\.3](https://arxiv.org/html/2607.06603#A7.SS3.p1.2),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.13.13.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.24.24.7),[Table G\.5](https://arxiv.org/html/2607.06603#A7.T5),[§1](https://arxiv.org/html/2607.06603#S1.p5.6),[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.06603#S3.T2.6.10.2.1),[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p1.4),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.23.12.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[37\]L\. Scimeca, S\. J\. Oh, S\. Chun, M\. Poli, and S\. Yun\(2022\)Which shortcut cues will dnns choose? a study from the parameter\-space perspective\.InInt\. Conf\. Learn\. Represent\. \(ICLR\),Cited by:[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p4.9)\.
- \[38\]J\. Song, C\. Meng, and S\. Ermon\(2020\)Denoising diffusion implicit models\.arXiv preprint arXiv:2010\.02502\.Cited by:[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p3.3)\.
- \[39\]M\. Wang, W\. Deng, J\. Hu, X\. Tao, and Y\. Huang\(2019\)Racial faces in the wild: reducing racial bias by information maximization adaptation network\.InInt\. Conf\. Comput\. Vis\. \(ICCV\),pp\. 692–702\.Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p1.1)\.
- \[40\]H\. Wu, G\. Bezold, M\. Günther, T\. Boult, M\. C\. King, and K\. W\. Bowyer\(2023\)Consistency and accuracy of celeba attribute values\.InIEEE Conf\. Comput\. Vis\. Pattern Recog\. \(CVPR\),pp\. 3257–3265\.Cited by:[§2](https://arxiv.org/html/2607.06603#S2.p5.1)\.
- \[41\]D\. Xu, S\. Yuan, L\. Zhang, and X\. Wu\(2018\)Fairgan: fairness\-aware generative adversarial networks\.In2018 IEEE International Conference on Big Data \(Big Data\),pp\. 570–575\.Cited by:[§2](https://arxiv.org/html/2607.06603#S2.p2.1)\.
- \[42\]M\. Yurochkin and Y\. Sun\(2020\)Sensei: sensitive set invariance for enforcing individual fairness\.arXiv preprint arXiv:2006\.14168\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p1.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.4.4.3),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.16.16.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.30.30.7),[§3\.1](https://arxiv.org/html/2607.06603#S3.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.06603#S3.T2.6.11.3.1)\.
- \[43\]M\. B\. Zafar, I\. Valera, M\. G\. Rogriguez, and K\. P\. Gummadi\(2017\)Fairness constraints: mechanisms for fair classification\.InInt\. Conf\. Artificial Intelligence and Statistics \(AISTATS\),pp\. 962–970\.Cited by:[§D\.3](https://arxiv.org/html/2607.06603#A4.SS3.p2.1),[Table D\.1](https://arxiv.org/html/2607.06603#A4.T1.8.8.3),[1\(a\)](https://arxiv.org/html/2607.06603#A7.T1.st1.25.25.4),[1\(b\)](https://arxiv.org/html/2607.06603#A7.T1.st2.51.51.7),[§1](https://arxiv.org/html/2607.06603#S1.p2.1),[Table 5](https://arxiv.org/html/2607.06603#S6.T5.11.15.4.1),[§6](https://arxiv.org/html/2607.06603#S6.p1.1)\.
- \[44\]F\. Zhang, K\. Kuang, L\. Chen, Y\. Liu, C\. Wu, and J\. Xiao\(2022\)Fairness\-aware contrastive learning with partially annotated sensitive attributes\.InInt\. Conf\. Learn\. Represent\. \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.06603#S1.p3.1),[§2](https://arxiv.org/html/2607.06603#S2.p2.1)\.
- \[45\]B\. Zi, S\. Zhao, X\. Ma, and Y\. Jiang\(2021\)Revisiting adversarial robustness distillation: robust soft labels make student better\.InInt\. Conf\. Comput\. Vis\. \(ICCV\),pp\. 16443–16452\.Cited by:[§5\.1](https://arxiv.org/html/2607.06603#S5.SS1.p3.3)\.

## Checklist

The checklist follows the references\. Please read the checklist guidelines carefully for information on how to answer these questions\. For each question, change the default\[TODO\]to\[Yes\],\[No\], or\[N/A\]\. You are strongly encouraged to include ajustification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description\. For example:

- •Did you include the license to the code and datasets?\[Yes\]See Section[H](https://arxiv.org/html/2607.06603#A8)\.
- •Did you include the license to the code and datasets?\[No\]The code and the data are proprietary\.
- •Did you include the license to the code and datasets?\[N/A\]

Please do not modify the questions and only use the provided macros for your answers\. Note that the Checklist section does not count towards the page limit\. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below\.

1. 1\.For all authors… 1. \(a\)Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?\[Yes\]Our claims are reflected accurately\. 2. \(b\)Did you describe the limitations of your work?\[Yes\]See[Appendix˜B](https://arxiv.org/html/2607.06603#A2) 3. \(c\)Did you discuss any potential negative societal impacts of your work?\[Yes\]See[Appendix˜B](https://arxiv.org/html/2607.06603#A2) 4. \(d\)Have you read the ethics review guidelines and ensured that your paper conforms to them?\[Yes\]
2. 2\.If you are including theoretical results… 1. \(a\)Did you state the full set of assumptions of all theoretical results?\[Yes\]See[Section˜4](https://arxiv.org/html/2607.06603#S4) 2. \(b\)Did you include complete proofs of all theoretical results?\[Yes\]See[Appendix˜A](https://arxiv.org/html/2607.06603#A1)
3. 3\.If you ran experiments \(e\.g\. for benchmarks\)… 1. \(a\)Did you include the code, data, and instructions needed to reproduce the main experimental results \(either in the supplemental material or as a URL\)?\[Yes\]See[Appendix˜D](https://arxiv.org/html/2607.06603#A4) 2. \(b\)Did you specify all the training details \(e\.g\., data splits, hyperparameters, how they were chosen\)?\[Yes\]See[Appendix˜D](https://arxiv.org/html/2607.06603#A4) 3. \(c\)Did you report error bars \(e\.g\., with respect to the random seed after running experiments multiple times\)?\[Yes\]See[Section˜G\.1](https://arxiv.org/html/2607.06603#A7.SS1) 4. \(d\)Did you include the total amount of compute and the type of resources used \(e\.g\., type of GPUs, internal cluster, or cloud provider\)?\[Yes\]See[Section˜D\.2](https://arxiv.org/html/2607.06603#A4.SS2)
4. 4\.If you are using existing assets \(e\.g\., code, data, models\) or curating/releasing new assets… 1. \(a\)If your work uses existing assets, did you cite the creators?\[Yes\]We citedLiuet al\.\[[25](https://arxiv.org/html/2607.06603#bib.bib98)\],Huanget al\.\[[13](https://arxiv.org/html/2607.06603#bib.bib20)\]andBrookset al\.\[[2](https://arxiv.org/html/2607.06603#bib.bib26)\]\. 2. \(b\)Did you mention the license of the assets?\[Yes\]See[Section˜C\.3](https://arxiv.org/html/2607.06603#A3.SS3) 3. \(c\)Did you include any new assets either in the supplemental material or as a URL?\[Yes\]See[Appendix˜H](https://arxiv.org/html/2607.06603#A8) 4. \(d\)Did you discuss whether and how consent was obtained from people whose data you’re using/curating?\[N/A\]We did not create new data but edited existing image benchmarks, so this issue does not apply\. 5. \(e\)Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?\[N/A\]We did not create new data but edited existing image benchmarks, so this issue does not apply\.
5. 5\.If you used crowdsourcing or conducted research with human subjects… 1. \(a\)Did you include the full text of instructions given to participants and screenshots, if applicable?\[Yes\]See[Figure˜C\.1](https://arxiv.org/html/2607.06603#A3.F1),[C\.2](https://arxiv.org/html/2607.06603#A3.F2), and[C\.3](https://arxiv.org/html/2607.06603#A3.F3)\. 2. \(b\)Did you describe any potential participant risks, with links to Institutional Review Board \(IRB\) approvals, if applicable?\[N/A\] 3. \(c\)Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?\[Yes\]See[Section˜C\.2](https://arxiv.org/html/2607.06603#A3.SS2)

## Appendix AProof of[Theorem˜4\.1](https://arxiv.org/html/2607.06603#S4.Thmtheorem1)

We start from LHS in equation[4\.1](https://arxiv.org/html/2607.06603#S4.Ex1):

\|P\(Y^=y′\|A=0,\\displaystyle\\big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|A=0,Y=y\)−P\(Y^=y′\|A=1,Y=y\)\|\\displaystyle Y=y\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|A=1,Y=y\)\\big\|=\|∑XA,XY,XGP\(Y^=\\displaystyle=\\bigg\|\\sum\_\{X\_\{A\},X\_\{Y\},X\_\{G\}\}P\(\\widehat\{Y\}=y′\|XA,XY,XG,A=0,Y=y\)P\(XA,XY,XG\|A=0,Y=y\)\\displaystyle y^\{\\prime\}\|X\_\{A\},X\_\{Y\},X\_\{G\},A=0,Y=y\)P\(X\_\{A\},X\_\{Y\},X\_\{G\}\|A=0,Y=y\)−P\(Y^=y′\|XA,XY,XG,A=1,Y=y\)P\(XA,XY,XG\|A=1,Y=y\)\|\\displaystyle\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{A\},X\_\{Y\},X\_\{G\},A=1,Y=y\)P\(X\_\{A\},X\_\{Y\},X\_\{G\}\|A=1,Y=y\)\\bigg\|\(A\.1\)=\|∑XA,XY,XGP\(Y^=y′\|XY,XG\)P\(XA,\\displaystyle=\\bigg\|\\sum\_\{X\_\{A\},X\_\{Y\},X\_\{G\}\}P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)P\(X\_\{A\},XY,XG\|A=0,Y=y\)\\displaystyle X\_\{Y\},X\_\{G\}\|A=0,Y=y\)−P\(Y^=y′\|XY,XG\)P\(XA,XY,XG\|A=1,Y=y\)\|\\displaystyle\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)P\(X\_\{A\},X\_\{Y\},X\_\{G\}\|A=1,Y=y\)\\bigg\|\(A\.2\)=\|∑XY,XGP\(Y^=y′\|XY,XG\)\(P\(XY\|XG\\displaystyle=\\bigg\|\\sum\_\{X\_\{Y\},X\_\{G\}\}P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\\big\(P\(X\_\{Y\}\|X\_\{G\},A=0,Y=y\)P\(XG\|A=0,Y=y\)\\displaystyle,A=0,Y=y\)P\(X\_\{G\}\|A=0,Y=y\)−P\(XY\|XG,A=1,Y=y\)P\(XG\|A=1,Y=y\)\)\|\\displaystyle\-P\(X\_\{Y\}\|X\_\{G\},A=1,Y=y\)P\(X\_\{G\}\|A=1,Y=y\)\\big\)\\bigg\|\(A\.3\)=\|∑XY,XGP\(Y^=y′\|XY,XG\)\(\\displaystyle=\\bigg\|\\sum\_\{X\_\{Y\},X\_\{G\}\}P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\\bigg\(P\(XY\|Y=y\)P\(XG\|A=0,Y=y\)−P\(XY\|Y=y\)P\(XG\|A=1,Y=y\)\)\|\\displaystyle P\(X\_\{Y\}\|Y=y\)P\(X\_\{G\}\|A=0,Y=y\)\-P\(X\_\{Y\}\|Y=y\)P\(X\_\{G\}\|A=1,Y=y\)\\bigg\)\\bigg\|\(A\.4\)=\|∑XYP\(XY\|Y=y\)\(∑XGP\(Y^\\displaystyle=\\bigg\|\\sum\_\{X\_\{Y\}\}P\(X\_\{Y\}\|Y=y\)\\Big\(\\sum\_\{X\_\{G\}\}P\(\\widehat\{Y\}=y′\|XY,XG\)P\(XG\|A=0,Y=y\)\\displaystyle=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)P\(X\_\{G\}\|A=0,Y=y\)\(A\.5\)−∑XG′P\(Y^=y′\|XY,XG′\)P\(XG′\|A=1,Y=y\)\)\|\.\\displaystyle\-\\sum\_\{X\_\{G\}^\{\\prime\}\}P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}^\{\\prime\}\)P\(X\_\{G\}^\{\\prime\}\|A=1,Y=y\)\\Big\)\\bigg\|\.\(A\.6\)Note the first and third equalities are driven by Bayes’ theorem, the second one is from the independence betweenY^\\widehat\{Y\}andXG,AX\_\{G\},Aconditioned onXY,XGX\_\{Y\},X\_\{G\}based on the Markov properties of SCM, and the fourth one is due to the independence betweenXYX\_\{Y\}andXG,AX\_\{G\},Aconditioned onYY\. We denote a coupling between the two distributionsP​\(XG\|A=0,Y\)P\(X\_\{G\}\|A=0,Y\)andP​\(XG′\|A=1,Y\)P\(X\_\{G\}^\{\\prime\}\|A=1,Y\)asΠ​\(XG,XG′\)\\Pi\(X\_\{G\},X\_\{G\}^\{\\prime\}\), then we have:

\|P\(Y^=y′\|A=0,Y=y\)−P\(Y^=y′\|A=1,Y=y\)\|\\displaystyle\\big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|A=0,Y=y\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|A=1,Y=y\)\\big\|=\|∑XYP\(XY\|Y=y\)\(∑XG,XG′Π\(XG,XG′\)\(P\(Y^=y′\|XY,XG\)−P\(Y^=y′\|XY,XG′\)\)\)\|\.\\displaystyle=\\bigg\|\\sum\_\{X\_\{Y\}\}P\(X\_\{Y\}\|Y=y\)\\Big\(\\sum\_\{X\_\{G\},X\_\{G\}^\{\\prime\}\}\\Pi\(X\_\{G\},X\_\{G\}^\{\\prime\}\)\\big\(P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}^\{\\prime\}\)\\big\)\\Big\)\\bigg\|\.\(A\.7\)≤∑XYP\(XY\|Y=y\)\(∑XG,XG′Π\(XG,XG′\)\|P\(Y^=y′\|XY,XG\)−P\(Y^=y′\|XY,XG′\)\|\)\\displaystyle\\leq\\sum\_\{X\_\{Y\}\}P\(X\_\{Y\}\|Y=y\)\\bigg\(\\sum\_\{X\_\{G\},X\_\{G\}^\{\\prime\}\}\\Pi\(X\_\{G\},X\_\{G\}^\{\\prime\}\)\\Big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}^\{\\prime\}\)\\Big\|\\bigg\)\(A\.8\)=∑XYP​\(XY\|Y=y\)​∑XG,XG′Π​\(XG,XG′\)​d𝜽,XY​\(XG,XG′\)\\displaystyle=\\sum\_\{X\_\{Y\}\}P\(X\_\{Y\}\|Y=y\)\\sum\_\{X\_\{G\},X\_\{G\}^\{\\prime\}\}\\Pi\(X\_\{G\},X\_\{G\}^\{\\prime\}\)d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X\_\{G\}^\{\\prime\}\)\(A\.9\)where the sample distance is denoted asd𝜽,XY\(XG,XG′\)=\|P\(Y^=y′\|XY,XG\)−P\(Y^=y′\|XY,XG′\)\|d\_\{\\bm\{\\theta\},X\_\{Y\}\}\(X\_\{G\},X\_\{G\}^\{\\prime\}\)=\\big\|P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}\)\-P\(\\widehat\{Y\}=y^\{\\prime\}\|X\_\{Y\},X\_\{G\}^\{\\prime\}\)\\big\|\. The inequality in Equation[A\.8](https://arxiv.org/html/2607.06603#A1.E8)is driven by Jensen’s inequality\.

![Refer to caption](https://arxiv.org/html/2607.06603v1/x4.png)

Figure A\.1:LFW\-CF examples\. The counterfactual \(CTF\) images regarding the “sex attribute” are shown\. The top row shows the original image, while the bottom row displays the CTF image generated by IP2P\.![Refer to caption](https://arxiv.org/html/2607.06603v1/x5.png)Figure A\.2:CF examples with other sensitive attributes\. Original and CTF samples are shown when age or skin color is considered as the sensitive attribute\.
## Appendix BLimitations and societal impacts

While our datasets and analyses reveal the relationship between CF and GF in image classification, we clarify our study’s limitations\. First of all, our study uses sex as a sensitive attribute based on visually perceived biological traits\. However, as mentioned in[Section˜2](https://arxiv.org/html/2607.06603#S2), this simplification does not capture the full spectrum of sexual traits, which is more complex and nuanced\. Therefore, we emphasize again that practitioners should use our data with these considerations in mind; they should not utilize our datasets for gender categorization but rather for investigating the unfairness in terms of CF and GF and enhancing fairness in AI systems\. Second, our data generation process relies on IP2P to create CTF samples\. We tried to mitigate the potential bias problem during the data generation process through the sophisticated human filtering process, but our data samples could be affected by the unintended bias of IP2P\. Third, while we assume the structural causal model \(SCM\) for images as[Figure˜2](https://arxiv.org/html/2607.06603#S4.F2), specifying an SCM in the real world is often infeasible\. This difficulty also makes it challenging to apply some of our experiments, such as analyzing the attributeGGor using a robust teacher model toGG\. However, we addressed these challenges to some extent by proposing a systematic method to investigateGG\([Section˜E\.2](https://arxiv.org/html/2607.06603#A5.SS2)\) and analyzing the robustness of vanilla teacher models \([Section˜5\.3](https://arxiv.org/html/2607.06603#S5.SS3)\)\.

Despite the limitations, we believe that our work makes significant contributions through our curated image datasets and extensive analyses to the evolving field that addresses relationships among different fairness notions\.

## Appendix CDatasets: CelebA\-CF and LFW\-CF

### C\.1Hyperparameters for IP2P image editing

We set the resolution of generated images to 256×\\times256 and the denoising step to 50\. Furthermore, we set the Image\-CFG weight to 1\.8 and the Text\-CFG weight to 7\.5\. These two hyperparameters are the guidance scales to control how the generated images closely resemble the input image or are intensely edited\. To alter the sensitive attribute of facial images, we use the prompts of “turn the woman into a man” for female images and “turn the man into a woman” for male images\.

### C\.2Human annotations

After generating CTF images using IP2P, we filtered them through five annotators to construct high\-quality CTF samples, namely CelebA\-CF and LFW\-CF\. Before evaluating the created CTF samples, the guidelines are given to the annotators, as presented in[Figure˜C\.2](https://arxiv.org/html/2607.06603#A3.F2)\. To establish the guidelines, we extracted 20 facial attributes of secondary sex characteristics using Chat\-GPT and then, with guidance from experts specialized in fairness, selected 9 key facial attributes \(facial hair, Adam’s apple, skin texture, jawline, chin shape, brow ridge, cheekbone prominence, lip fullness, and hairline\)\. The guidelines instruct human annotators to filter out counterfactual samples based on these attributes \(including considerations for the presence of makeup\)\. Subsequently, given the original image and generated CTF image pair, annotators assess whether the generated image is correctly created based on the instructions\.[Figure˜C\.1](https://arxiv.org/html/2607.06603#A3.F1)shows the interfaces of the annotation task for image filtering\.

We further verify the reliability of our datasets with another five human annotators, different from those who participated in the previous filtering process, and report the result in[Table˜1](https://arxiv.org/html/2607.06603#S2.T1)\. For more objective annotation, we show 8 example images to the annotators before the labeling task, which are randomly sampled the same number of times for each attribute value from test image datasets\. Then, the annotators label CelebA\-CF and LFW\-CF for 4 attributes,*i\.e\.*, “sex”, “blond hair”, “gray hair”, and “smiling” in order\. Specifically, the annotators evaluate whether the sensitive attribute was correctly altered and the non\-sensitive attributes were maintained for a generated image\. Similar to the image filtering task, we provide the annotators with the set of 10 sex\-related facial attributes for objective and accurate labeling\.[Figure˜C\.3](https://arxiv.org/html/2607.06603#A3.F3)shows the interfaces of the annotation task for this reliability check\. We provide the attribute values originally annotated for the original image datasets on the screen together for annotators to refer to as a guide for their annotating tasks\. Although we focus on visually perceived sex traits, we use the terms Male and Female for convenience in the annotation interface\.

We note that the wage paid to each participant is 18 USD per hour, resulting in a total expenditure of 360 USD on participant compensation\.

![Refer to caption](https://arxiv.org/html/2607.06603v1/x6.png)Figure C\.1:User Interface shown to five annotators for image filtering\.![Refer to caption](https://arxiv.org/html/2607.06603v1/x7.png)Figure C\.2:The guideline instructions that were given to the five annotators for the image filtering\.![Refer to caption](https://arxiv.org/html/2607.06603v1/x8.png)Figure C\.3:User Interface shown to five annotators to evaluate the reliability of our created datasets\.
### C\.3License information of assets employed in this study

- •
- •LFW\[[13](https://arxiv.org/html/2607.06603#bib.bib20)\]is publicly available for research purposes\. There is also no formal license and further information is reported at[https://vis\-www\.cs\.umass\.edu/lfw/](https://vis-www.cs.umass.edu/lfw/)\.
- •
- •

### C\.4Further information of the new dataset\.

CelebA\-CF and LFW\-CF are based on real facial image datasets, such as CelebA and LFW, which include 40 attribute annotations\. We note that because the original datasets have an imbalance between some pairs of two attributes, our datasets also possess a different skewness between the sensitive attribute and other attributes\. For example, in CelebA and CelebA\-CF, most images with the blond hair attribute are female, and most males do not have blond hair\. We present the skewness values between the sensitive attribute and other non\-sensitive attributes in CelebA\-CF in[Table˜C\.1](https://arxiv.org/html/2607.06603#A3.T1)\. Additionally, we displayed the group\-specific failure rate identified through the filtering process in[Table˜C\.2](https://arxiv.org/html/2607.06603#A3.T2)\.

Table C\.1:The skewness between the sensitive attribute and other non\-sensitive attributes in CelebA\-CF\.Table C\.2:The failure rate identified through the filtering process\.This table shows the proportion of images filtered out, calculated separately for each sensitive attribute \(*e\.g\.*, female, male\), in constructing the CelebA\-CF and LFW\-CF datasets\. The sensitive attributes in the table represent the labels of the original images\.

## Appendix DImplementation details

### D\.1Details on training datasets

For CelebA, we utilize the official train\-validation\-test split\[[25](https://arxiv.org/html/2607.06603#bib.bib98)\]\. For LFW, we also use the official train\-test split\[[13](https://arxiv.org/html/2607.06603#bib.bib20)\]and then divide the training data into a training and a validation set, with a ratio of 80:20\.

CIFAR\-10B is a modified dataset from CIFAR\-10, as described in[Section˜5\.2](https://arxiv.org/html/2607.06603#S5.SS2)\. We modify CIFAR\-10 into a binary classification task by dividing the original 10 classes into two classes \(classes 0\-4 and 5\-9\)\. To introduce a fairness issue, we set the sensitive attributeAAas the presence of Gaussian noise and skew the dataset by randomly injecting the noise into 20% and 80% of the data in class 0 and class 1, respectively\. Additionally, we introduce Contrast noise for the attributeGG\. Using the skew\-ratioα\\alpha, we create a statistical correlation betweenAAandGGby adding the noise into100×α100\\times\\alpha% of the data samples withA=1A=1and100×\(1−α\)100\\times\(1\-\\alpha\)% of the data samples withA=0A=0\. Unless otherwise noted, we setα\\alphato 0\.8\. We partition the dataset into train\-validation\-test sets with a ratio of 64:16:20, respectively, maintaining consistent values for the two skewness ratios \(*i\.e\.*, skewness betweenAAandYY,GGandAA\) across all sets during our experiments\.

### D\.2Compute Infrastructure and optimization

Our all experiments including the dataset construction and performance comparison of existing methods and CKD were conducted using AMD Ryzen Threadripper PRO 3975WX CPUs and NVIDIA RTX A5000 GPUs\. Our dataset generation was parallelized using 8 GPUs and took 2 days to complete\. Training time for models used in the experiments for performance comparison ranges between 12 to 24 hours depending on the dataset and method used\.

For CIFAR\-10B, we use ResNet56 models with the Adam optimizer for 50 epochs\. We set the mini\-batch size and learning rate as 128 and 0\.001, respectively\. Because the skewness betweenGGandAAin CIFAR\-10B test datasets varies, we compute the balanced CD over both the target class and the sensitive attribute for the consistent metric\. For CelebA and LFW, we train ResNet18 models with the AdamW optimizer\. We use the epoch size of 70 and 50 for each dataset, and set the mini\-batch size, learning rate, and weight decay as 128, 0\.001, and 1e\-4, respectively\. We use identical hyperparameters regarding the optimization for all methods\. All results are averaged over results from four different random seeds\.

### D\.3Implementation details of baselines and CKD

CF\-aware methods\.Scratch \(\+Aug\)\[[7](https://arxiv.org/html/2607.06603#bib.bib176)\]minimizes the empirical cross\-entropy loss computed using both original and counterfactual images\. CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]has a regularization term that promotes the image pairs to be the same prediction\. We use logits of a neural network model as representation vectors for the CP regularization term\. Since Scratch \(\+aug\) and CP utilize CTF samples, we generated these samples using IP2P with the same prompt in[Section˜C\.1](https://arxiv.org/html/2607.06603#A3.SS1)using the image\-CFG of 7\.5 and the Text\-CFG of 2\.0\. We note that we do not apply any filtering process for their generated training datasets\. SenSeI\[[42](https://arxiv.org/html/2607.06603#bib.bib25)\]uses two metrics for training: one for a pre\-defined fair regularizer distance metric and the other obtained by fair metric learning\. We use the same metrics as presented in their code\. By generating the worst\-case samples based on these metrics, we apply a fair regularization term to promote their predictions to be the same, as originally implemented\. LASSI\[[31](https://arxiv.org/html/2607.06603#bib.bib24)\]minimizes an objective function which is composed of the classification loss, the reconstruction loss, and the adversarial loss to learn individually fair representation\. We use the official code of LASSI as it is\.

GF\-aware methods\.LBC\[[15](https://arxiv.org/html/2607.06603#bib.bib89)\]necessitates multiple full\-training iterations, alternately re\-weighting each group based on the given group fairness metric and re\-training\. Due to its high computation budget for iterative full\-training, we limit the number of epochs for each training to 5 and repeat this process 14 times\. COV\[[43](https://arxiv.org/html/2607.06603#bib.bib83)\]utilizes a fairness constraint based on the covariance between the group label and the signed distance of feature vectors from the decision boundary of a classifier\. We minimize the constraint\-regularized objective function through gradient descent optimization, instead of directly solving its optimization problem\. MFD\[[16](https://arxiv.org/html/2607.06603#bib.bib86)\]employs an additional fairness\-promoting regularization term based on Maximum Mean Discrepancy \(MMD\)\. For the MMD distance of the regularization term, we use the Gaussian RBF kernel with the variance parameter set as the mean of squared distance between all data points\. We implemented SS\[[14](https://arxiv.org/html/2607.06603#bib.bib58)\]and RW\[[17](https://arxiv.org/html/2607.06603#bib.bib82)\]identically to the original algorithm\.

CF\- and GF\-aware methods\.The combinations of GF method and the augmentation were implemented so that GF methods train a model on their own objective function using training datasets augmented by generated CTF samples\. The combinations of GF methods and CP optimize the objective functions of GF methods combined by the CP regularization\. The CKD regularization term \([5\.1](https://arxiv.org/html/2607.06603#S5.Ex3)\) builds upon representation vectorsf​\(𝜽,x\)f\(\\bm\{\\theta\},x\)\. For this vector, we use the logits of a neural network model on LFW and CIFAR\-10B\. For CelebA, we utilize feature vectors from the penultimate layer of models as a representation vector since its training dataset is relatively much larger and more complex than others, leading to more fine\-grained feature vectors\.

### D\.4Hyperparameter search

The range of hyperparameter search used for[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)and[Table˜H\.1](https://arxiv.org/html/2607.06603#A8.T1)are shown in[Table˜D\.1](https://arxiv.org/html/2607.06603#A4.T1)\. We utilize grid search to select hyperparameter values within a certain range\. Note CKD and the combinations of GF\-aware methods and CP have additional parametersλ\\lambdafor CP loss\. We use the same range as CP forλ\\lambda\. We also note that for LASSI, we only search the hyperparameter for an adversarial loss while maintaining other parameters as the same as used in their experiments on CelebA\.

Table D\.1:Hyperparameters and search ranges for each method\.

## Appendix ERate of Flipped Predictions \(RFP\)

### E\.1RFP measurement on CelebA

To measure RFP on CelebA, we assume that the hair length of facial images isGG,*i\.e\.*, it is correlated with, but not caused by, the sensitive attribute\. Then, we use Stochastic Differential Editing \(SDEdit\)\[[28](https://arxiv.org/html/2607.06603#bib.bib29)\], an image editing method based on a diffusion model, to modify the hair length in each image\. SDEdit selectively edits specific regions of a given image based on the colored stroke\. Namely, SDEdit depicts the image region indicated by the stroke with the given color in the most plausible manner\. By doing so, SDEdit generates realistic and faithful edited images, while preventing changes in the region not indicated by the strokes\. To utilize SDEdit, we randomly select 40 samples for each group of the same target label and sensitive attribute from original samples of CelebA\-CTF pairs and then we manually apply strokes on the hair of facial images for a total of 160 samples\. Specifically, To extend the hair length, we applied strokes with the hair color to the areas where the hair should grow\. Conversely, to shorten the hair length, we applied strokes with the background color to the areas where the hair should be removed\. After this process, we utilize the official PyTorch implementation ofMenget al\.\[[28](https://arxiv.org/html/2607.06603#bib.bib29)\]to edit images with the applied strokes\.[Figure˜E\.1](https://arxiv.org/html/2607.06603#A5.F1)shows some examples of images edited by SDEdit\.

### E\.2Discussion about selectingGGon CelebA

As mentioned in[Section˜5\.2](https://arxiv.org/html/2607.06603#S5.SS2), we intuitively chose “hair length” asGGon CelebA sinceGGis highly correlated with, but not caused by, the sensitive attributeAA\. However, we introduce a more generalized approach for choosingGGby leveraging a CP\-trained model that exhibits low CD but high DEO\. Specifically, using a pre\-defined set of attributes, we first train a linear classifier on the top of the feature extractor from the CP\-trained model for each attribute\. In cases where annotations are not available, CLIP\-based pseudo labels can be utilized\. Based on the accuracy of each linear classifier, we can then identify which attributes the CP\-trained model learns more, indicating potential heavy reliance on these attributes\. Finally, we can selectGGattributes based on two criteria: \(1\) high accuracy of a linear classifier and \(2\) high correlation \(not causation\) with the sensitive attribute\.

To validate this approach, we conducted an experiment on CelebA dataset using a subset of 40 pre\-annotated attributes and the “hair length” \(”hair length” labels are predicted by CLIP as it is not originally labeled in CelebA\)\.[Table˜E\.1](https://arxiv.org/html/2607.06603#A5.T1)displays the accuracy for each attribute after training a linear classifier on the top of the CP\-trained model, alongside the skewness between the sensitive attribute and other attributes\. Attributes like “Pale Skin” show a high correlation with the sensitive attribute but low accuracies, suggesting CP might not rely on them \(not satisfying the second condition\)\. “Mouth Sightly Open” exhibits low correlation and accuracy, thus not being considered asGG\(failing the first condition\)\. In contrast, attributes such as “bangs”, “wearing hat”, and “hair length” exhibit both high correlation values and accuracies, indicating that they are promising candidates for the attributeGG\.

Table E\.1:The skewness between the sensitive attribute and other attributes, as well as the accuracy for each attribute after re\-training a linear classifier on the top of the CP\-trained model\.![Refer to caption](https://arxiv.org/html/2607.06603v1/x9.png)

Figure E\.1:Examples of CelebA measuring RFP with respect to “hair length”\.If the person in the original image \(first row\) had long hair, we create a modified image \(second row\) with shorter hair and vice versa\.![Refer to caption](https://arxiv.org/html/2607.06603v1/x10.png)

Figure E\.2:Illustration of CIFAR\-10B\.The sensitive attributeAAis characterized by the line type \(A=0A=0for a solid line,A=1A=1for a dashed line\), while another attributeGGis denoted by the color of the line \(G=0G=0for a red line,G=1G=1for a green line\)\.

## Appendix FFurther discussion about the faithfulness assumption

The faithful assumption states that if some two variables are statistically independent, they are d\-separated,*i\.e\.*, there is no connected path between them\. Thus, under this assumption, GF\-aware methods, which enforce independence between the sensitive attribute and the target label, can achieve CF simultaneously when they successfully achieve GF\. The previous works\[[1](https://arxiv.org/html/2607.06603#bib.bib51),[35](https://arxiv.org/html/2607.06603#bib.bib9)\]provide the same argument that GF implies CF under the faithfulness assumption\. However, some other previous works\[[18](https://arxiv.org/html/2607.06603#bib.bib130)\]have demonstrated that while the faithfulness assumption is crucial for causal inference literature, it may not always hold true, especially in complex real\-world scenarios\. Moreover, the result for GF\-aware methods in[Table˜H\.1](https://arxiv.org/html/2607.06603#A8.T1)reveals that the GF\-aware methods have a minimal impact on improving CD, implying that the faithfulness assumption does not hold in CelebA and LFW datasets\.

## Appendix GAdditional results

Table G\.1:Evaluation of GF and CF of fair\-training for image classification\. We put the results from[Table˜2](https://arxiv.org/html/2607.06603#S3.T2)and[Table˜H\.1](https://arxiv.org/html/2607.06603#A8.T1)together with the standard deviation values over four different seeds\.\(a\)Standard deviations on CIFAR\-10B\.
\(b\)Standard deviations on CelebA and LFW\.

Table G\.2:Impact of robustness toGGof the teacher model on CKD with Celeb and LFW\.𝜽CKDT\\bm\{\\theta\}^\{T\}\_\{\\text\{CKD\}\}and𝜽CPT\\bm\{\\theta\}^\{T\}\_\{\\text\{CP\}\}are CKD and CP trained teacher models\.𝜽ScratchT\\bm\{\\theta\}^\{T\}\_\{\\text\{Scratch\}\}is a vanilla\-trained teacher model\. RFPTdenotes how a teacher is biased towardsGG\. DEO, CD, RFP are metrics for evaluating GF, CF, and bias towardsGG, respectively\. Since we generate a dataset for RFP measurements on CelebA with “hair length” asGG, we report RFP values only for CelebA\.### G\.1Result tables with standard deviation

We report the standard deviation values of the performance comparison results in[Table˜G\.1](https://arxiv.org/html/2607.06603#A7.T1)for CIFAR\-10B, CelebA, and LFW respectively\. The standard deviation values are calculated over four different seeds\.

### G\.2Impact of the robustness toGGof the teacher model on CKD with CelebA and LFW

We analyze the effectiveness of the teacher model on real image datasets\. We report the performance of CKD using variants of teacher models that are more or less robust toGGon CelebA and LFW\. We consider three teacher models, ordered by robustness toGG: CP \(𝜽CPT\\bm\{\\theta\}^\{T\}\_\{\\text\{CP\}\}\), Scratch \(𝜽ScratchT\\bm\{\\theta\}^\{T\}\_\{\\text\{Scratch\}\}\), and CKD model with a Scratch teacher \(𝜽CKDT\\bm\{\\theta\}^\{T\}\_\{\\text\{CKD\}\}\)\.[Table˜G\.2](https://arxiv.org/html/2607.06603#A7.T2)displays ACC, DEO, CD \(on CelebA and LFW\), and RFP \(on CelebA\) depending on the teacher models\. Since we generate a dataset for RFP measurements on CelebA with “hair length” asGG, we report RFP values only for CelebA\. Through the result, we observe that compared to vanilla\-trained teachers𝜽ScratchT\\bm\{\\theta\}^\{T\}\_\{\\text\{Scratch\}\}, using more robust teachers \(*e\.g\.*,𝜽CKDT\\bm\{\\theta\}^\{T\}\_\{\\text\{CKD\}\}\) achieves slightly better or competitive DEO and CD, while employing less robust teachers \(*e\.g\.*,𝜽CPT\\bm\{\\theta\}^\{T\}\_\{\\text\{CP\}\}\) significantly degrades DEO, which are consistent with the results in[Section˜5\.3](https://arxiv.org/html/2607.06603#S5.SS3)on CIFAR\-10B\.

### G\.3Analaysis on CKD

Ablation study\.To study the effectiveness of our CKD regularization term, we additionally consider a method that is a naive combination of a typical KD method proposed byHintonet al\.\[[12](https://arxiv.org/html/2607.06603#bib.bib137)\]and CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]\(*i\.e\.*, HKD\+CP\)\. Note that this combination can be considered as a baseline method that considers both CF and GF if it uses a robust teacher model toGG, because the method robustifies the model with respect toGGwhile achieving CF\.[Table˜G\.5](https://arxiv.org/html/2607.06603#A7.T5)compares the method with our CKD\. As we expected, the results show that HKD\+CP improves both DEO and CD simultaneously\. However, its performance is still suboptimal compared to CKD, showing CKD is more effective than the naive combination of KD and CP\.

CKD on feature vectors vs logits\.CKD can utilize either feature or logit vectors as the target vectors, i\.e\.,f​\(θ,x\)f\(\\theta,x\)\. For CIFAR\-10B and LFW, where we use logits as the target vectors in our experiments, we displayed the performance of CKD using feature vectors as the target vectors in[Table˜G\.6](https://arxiv.org/html/2607.06603#A7.T6)\. The results demonstrate that CKD using feature vectors exhibits comparable performance to those with logits\. Moreover, we can get even better performance on CIFAR\-10B using feature vectors, demonstrating that CF and GF can be achieved simultaneously regardless of which type of target vectors is used\.

Sensitiveness ofμ\\mu\.μ\\muis the regularization strength for the CKD loss term\. Specifically, asμ\\muincreases, we expect improvements in both DEO and CD\.[Table˜G\.7](https://arxiv.org/html/2607.06603#A7.T7)shows the performance of CKD across different values ofμ\\mu, aligning with our expectations\. Additionally, we note that CKD performance is insensitive toμ\\mu\.

The implication of using non\-curated IP2P\.Uncurated CTF datasets are imperfect\. Specifically, some samples generated from the original images in our test datasets were filtered out because the images either showed minimal changes or had alterations that affected non\-sensitive attributes including the target attributes\. Consequently, the more such incomplete samples exist, the more they will negatively impact the performance of our method\. To assess how sensitive CKD is to incomplete CTF training samples, we conducted additional experiments on CIFAR\-10B by varying the ratio of incomplete CTF samples in the training set\. For a given ratioα\\alpha, we assumed that half of the incomplete samples are nearly unchanged, while the other half are samples where both the target and sensitive attributes are altered\. We variedα\\alphafrom 20% to 60% in 10% increments and reported the accuracy, CD, and DEO of CKD in the table below\. The results in[Table˜G\.3](https://arxiv.org/html/2607.06603#A7.T3)indicate that CKD significantly improves both CD and DEO compared to Scratch, even for highα\\alphas\. Although this phenomenon has not been fully explained, we hypothesize that the robustness can be attributed to the distillation process, as empirically demonstrated in\[[8](https://arxiv.org/html/2607.06603#bib.bib39)\]\.

### G\.4Additional experimental results on counterfactual samples\.

We additionally report the accuracy \(acc\-CTF\) and DEO \(DEO\-CTF\) for CKD and several baseline methods on counterfactual samples in CelebA\-CF in[Table˜G\.4](https://arxiv.org/html/2607.06603#A7.T4)\.

### G\.5Additional experimental results for other metrics proposed by\[[33](https://arxiv.org/html/2607.06603#bib.bib183)\]\.

We first note that the Counterfactual Disparity \(CD\) we used is the same metric as the Switch Rate \(SR\) proposed by\[[33](https://arxiv.org/html/2607.06603#bib.bib183)\]\. We computed P2NR \(another metric proposed by\[[33](https://arxiv.org/html/2607.06603#bib.bib183)\]\) on CelebA and obtained values of0\.0360\.036,0\.1650\.165, and0\.3390\.339for Scratch, CP, and CKD, respectively\. These results indicate that CKD achieves low CD with a balanced rate of misclassification across the labels\. Additionally, we would like to emphasize that[Pintoet al\.](https://arxiv.org/html/2607.06603#bib.bib183)focused on scenarios where GF does not imply CF in their experiments—highlighting cases where the faithfulness assumption, which can be overly stringent, does not hold \(see line 323 in their paper\)\. However, our work primarily explores the converse: whether CF can imply GF depending on the presence ofGG, independent of the faithfulness assumption\.

Table G\.3:The implication of using non\-curated IP2P\.Table G\.4:The accuracy and DEO on counterfactual samples in CelebA\-CF\.Table G\.5:Evaluation of group fairness \(GF\) and counterfactual fairness \(CF\) of fair\-training for image classification\. The details are the same as[Table˜H\.1](https://arxiv.org/html/2607.06603#A8.T1)\. “HKD\+CP” denotes a model that naively combines Knowledge Distillation\[[12](https://arxiv.org/html/2607.06603#bib.bib137)\]with CP\[[36](https://arxiv.org/html/2607.06603#bib.bib8)\]\.Table G\.6:CKD on feature vectors vs logits\.Table G\.7:Sensitivity ofμ\\mu\.

## Appendix HDatasheet for dataset

### H\.1Motivation

For what purpose was the dataset created?These datasets were created for evaluating counterfactual fairness in image classifiers\. Furthermore, since our datasets contain counterfactual images generated from real\-world images, our datasets can be also used for analyzing the relationship between counterfactual and group fairness on image datasets\. For more discussion of the motivation behind our datasets, see[Section˜1](https://arxiv.org/html/2607.06603#S1)\.

Who created the dataset \(e\.g\., which team, research group\) and on behalf of which entity \(e\.g\., company, institution, organization\)?The datasets were created by the authors of this paper who were affiliated with Seoul National University and NAVER AI LAB\.

Who funded the creation of the dataset?Funding was provided by the National Research Foundation of Korea \(NRF\); Institute of Information & Communications Technology Planning & Evaluation \(IITP\); and the SNU\-Naver Hyperscale AI Center\.

### H\.2Composition

What do the instances that comprise the dataset represent \(e\.g\., documents, photos, people, countries\)?The instances represent synthetically generated images and corresponding real\-world original images from two popular benchmark facial image datasets, CelebA\[[25](https://arxiv.org/html/2607.06603#bib.bib98)\]and LFW\[[13](https://arxiv.org/html/2607.06603#bib.bib20)\]\.

How many instances are there in total \(of each type, if appropriate\)?CelebA\-CF and LFW\-CF contain a total of 230 and 144 image pairs of original and counterfactual images, respectively\.

Does the dataset contain all possible instances or is it a sample \(not necessarily random\) of instances from a larger set?We uniformly sampled a subset of test images in CelebA and LFW to balance the target and group labels \(see more details in[Section˜2](https://arxiv.org/html/2607.06603#S2)\)\. Then, we made our datasets including all possible samples according to our filtering process\.

What data does each instance consist of?Each instance contains a pair of original and counterfactual images\.

Is there a label or target associated with each instance?Yes, there are 40 binary annotations that originated from CelebA and LFW\.

Is any information missing from individual instances?No

Are relationships between individual instances made explicit \(e\.g\., users’ movie ratings, social network links\)?Yes, instances that correspond to a counterfactual pair are explicitly annotated as such in our dataset\. Otherwise, there are no relationships between individual instances\.

Are there recommended data splits \(e\.g\., training, development/validation, testing\)?No, the dataset is created for the purpose of testing\.

Are there any errors, sources of noise, or redundancies in the dataset?Using IP2P as the initial step in constructing our datasets might introduce some noise or errors in the datasets\. Refer to[Appendix˜B](https://arxiv.org/html/2607.06603#A2)for further details\.

Is the dataset self\-contained, or does it link to or otherwise rely on external resources \(e\.g\., websites, tweets, other datasets\)?Yes, it is self\-contained\.

Does the dataset contain data that might be considered confidential \(e\.g\., data that is protected by legal privilege or by doctor patient confidentiality, data that includes the content of individuals’ non\-public communications\)?No

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?Yes, the dataset may cause some anxiety about sex labels\. See[Section˜2](https://arxiv.org/html/2607.06603#S2)and[Appendix˜B](https://arxiv.org/html/2607.06603#A2)\.

Does the dataset identify any subpopulations \(e\.g\., by age, gender\)?Yes, our datasets were created after evaluating whether counterfactual samples regarding visually perceived sexual traits were generated correctly or not\. This evaluation was conducted by five human annotators\. Thus, our datasets contain the identification of visually perceived sexual traits which represent some statistically representative features for each sex\. See more discussion in[Section˜2](https://arxiv.org/html/2607.06603#S2)\.

Is it possible to identify individuals \(i\.e\., one or more natural persons\), either directly or indirectly \(i\.e\., in combination with other data\) from the dataset?Yes, our datasets are generated from CelebA and LFW, which are facial datasets collected on the internet\.

Does the dataset contain data that might be considered sensitive in any way \(e\.g\., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history\)?Yes, we set sex as the sensitive attribute and created our CTF samples with the sensitive attribute flipped\.

### H\.3Collection Process

How was the data associated with each instance acquired?Our datasets are generated through image editing using IP2P \(See[Section˜2](https://arxiv.org/html/2607.06603#S2)\)\.

What mechanisms or procedures were used to collect the data \(e\.g\., hardware apparatuses or sensors, manual human curation, software programs, software APIs\)?Refer to[Section˜2](https://arxiv.org/html/2607.06603#S2)for a complete description of our data generation process\.

If the dataset is a sample from a larger set, what was the sampling strategy \(e\.g\., deterministic, probabilistic with specific sampling probabilities\)?Original test samples in our datasets were uniformly sampled from the test datasets of CelebA and LFW\.

Who was involved in the data collection process \(e\.g\., students, crowdworkers, contractors\) and how were they compensated \(e\.g\., how much were crowdworkers paid\)?The filtering process for our datasets involved five student annotators who received about 18 USD per hour for their wage\.

Over what timeframe was the data collected?Our datasets were generated and evaluated over one month\.

Were any ethical review processes conducted \(e\.g\., by an institutional review board\)?No

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources \(e\.g\., websites\)?No, we initially obtained the data from publicly available sources\. Subsequently, we edited the data and filtered the edited one through human annotators\.

Were the individuals in question notified about the data collection?Not applicable

Did the individuals in question consent to the collection and use of their data?Not applicable

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?Not applicable

Has an analysis of the potential impact of the dataset and its use on data subjects \(e\.g\., a data protection impact analysis\) been conducted?Not applicable

### H\.4Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done \(e\.g\., discretization or bucketing, tokenization, part\-of\-speech tagging, SIFT feature extraction, removal of instances, processing of missing values\)?Yes, we filtered our generated datasets with human annotators\. Refer to[Section˜2](https://arxiv.org/html/2607.06603#S2)for a complete description of our filtering process\.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data \(e\.g\., to support unanticipated future uses\)?No, however, raw data can be reproduced by applying IP2P as described in[Section˜2](https://arxiv.org/html/2607.06603#S2)\.

Is the software that was used to preprocess/clean/label the data available?Yes, refer to the[Section˜2](https://arxiv.org/html/2607.06603#S2)\.

### H\.5Uses

Has the dataset been used for any tasks already?Yes, we applied our datasets to evaluate CF in image classifiers in[Section˜3](https://arxiv.org/html/2607.06603#S3)and[6](https://arxiv.org/html/2607.06603#S6)and analyze the relationship between CF and GF in[Section˜5](https://arxiv.org/html/2607.06603#S5)\.

Is there a repository that links to any or all papers or systems that use the dataset?We will provide a link to a repository on GitHub that includes references to all papers and systems utilizing the dataset\.

What \(other\) tasks could the dataset be used for?There is no other task where our dataset can be used\. The dataset is exclusively designed for evaluating counterfactual fairness in real\-world image datasets\.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?Because our datasets were generated through the image editing technique, IP2P\[[2](https://arxiv.org/html/2607.06603#bib.bib26)\], they may contain implicit biases or errors, which are present in the IP2P model\[[27](https://arxiv.org/html/2607.06603#bib.bib27),Bianchi\_2023\]\. While we have conducted a thorough human filtering and validation process to minimize these issues in our dataset, future users should still be aware of these limitations\.

Are there tasks for which the dataset should not be used?The dataset should not be employed for tasks where the limitations discussed in[Appendix˜B](https://arxiv.org/html/2607.06603#A2)could pose critical issues, or for tasks that are not for research purposes\.

### H\.6Distribution

Will the dataset be distributed to third parties outside of the entity \(e\.g\., company, institution, organization\) on behalf of which the dataset was created?Yes, the datasets will be made publicly available\.

How will the dataset will be distributed \(e\.g\., tarball on website, API, GitHub\)?The dataset will be distributed using tarball on the website\. Refer to[Appendix˜C](https://arxiv.org/html/2607.06603#A3)\.

When will the dataset be distributed?The datasets will be made publicly available upon acceptance\.

Will the dataset be distributed under a copyright or other intellectual property \(IP\) license, and/or under applicable terms of use \(ToU\)?The datasets will be distributed under the CC BY 4\.0 license\.

Have any third parties imposed IP\-based or other restrictions on the data associated with the instances?No

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?No

### H\.7Maintenance

Who will be supporting/hosting/maintaining the dataset?The datasets are hosted, supported, and maintained by the authors\.

How can the owner/curator/manager of the dataset be contacted \(e\.g\., email address\)?The corresponding author can be contacted by the e\-mail address which will be listed on the first page of this paper after camera\-ready\.

Is there an erratum?No

Will the dataset be updated \(e\.g\., to correct labeling errors, add new instances, delete instances\)?No future updates are currently planned\. However, we will monitor the GitHub repository for related issues and address any problems that arise\.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances \(e\.g\., were the individuals in question told that their data would be retained for a fixed period of time and then deleted\)?Not applicable

Will older versions of the dataset continue to be supported/hosted/maintained?Yes, if the datasets are updated, we will maintain the older versions\.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?Yes, we make our code and datasets public, and hence others can contribute or extend to our work and datasets\.

Table H\.1:Evaluation of GF and CF of fair\-training for image classification\.