From Model to Data (M2D): Shifting Complexity from GNNs to Graphs for Transparent Graph Learning

# From Model to Data (M2D): Shifting Complexity from GNNs to Graphs for Transparent Graph Learning
Source: [https://arxiv.org/html/2605.06814](https://arxiv.org/html/2605.06814)
Debolina Halder Lina Department of Computer Science, Rice University, Houston, TX 77005, USA Arlei Silva Department of Computer Science & Ken Kennedy Institute, Rice University, Houston, TX 77005, USA

###### Abstract

Graph Neural Networks \(GNNs\) achieve high performance but can be opaque to humans, making it difficult to understand and compare the many proposed architectures\. While existing explainability methods attribute individual predictions to nodes, edges, or features, they do not provide architectural transparency or explain the fundamental performance gap between simple and more complex models\. To address this limitation, we introduce Model\-to\-Data \(M2D\) distillation, a new framework that increases transparency by transferring model complexity into the data space\. M2D distills the teacher model into an augmented graph with enriched features and structure, enabling a simple student to match the teacher’s performance\. By materializing model behavior in the data, our approach allows humans to inspect architectural advantages directly\. We show that M2D reveals underlying mechanisms such as fairness objectives and attention\-based aggregation in an interpretable way, enhancing GNN transparency while preserving performance\.

## 1Introduction

Graph Neural Networks \(GNNs\) have achieved state\-of\-the\-art performance on graph tasks such as node classification, link prediction, and graph classification\(Gilmeret al\.,[2017](https://arxiv.org/html/2605.06814#bib.bib1); Kipf and Welling,[2017](https://arxiv.org/html/2605.06814#bib.bib2); Veličkovićet al\.,[2018](https://arxiv.org/html/2605.06814#bib.bib60); Hamiltonet al\.,[2017](https://arxiv.org/html/2605.06814#bib.bib3)\)\. However, as GNNs become more complex to address limitations such as biased predictions, lack of robustness to data perturbations, over\-smoothing, and limited expressiveness, they also become increasingly opaque to humans\. Thisepistemic opacityhas practical implications ranging from limiting the adoption of GNNs in high\-stakes settings to hindering our ability to understand why certain architectures outperform others on specific benchmarks\. Our work aims to make GNNs more transparent by enabling humans to \(1\) understand their logic and \(2\) audit them for correctness and biasLipton \([2018](https://arxiv.org/html/2605.06814#bib.bib38)\)\.

While traditional explainers identify which input features influenced a prediction, they fail to explain the architectural suitability of one model over another for a specific dataset\. For instance, it is unclear how the self\-attention mechanism translates to the performance gains of a Graph Attention Network \(GAT\) over a Graph Convolutional Network \(GCN\) on the Cora datasetVeličkovićet al\.\([2018](https://arxiv.org/html/2605.06814#bib.bib60)\)\. If we can materialize this architectural advantage into static data augmentations, we enable humans to audit the model’s logic directly within the data space, revealing why a complex architecture outperforms a simpler one\.

To this end, we introduce Model\-to\-Data \(M2D\) distillation, which transfers complexity from a teacher modelℳ\\mathcal\{M\}into an augmented graph𝔻\\mathbb\{D\}characterized by enriched node features and modified adjacency weights\. This allows a lightweight studentmmto achieve teacher\-level performance while maintaining a human\-auditable data representation\. We view M2D in a unified space of knowledge and data distillation, illustrated in Fig\.[2](https://arxiv.org/html/2605.06814#S1.F2)\. While Knowledge \(K\) distillationGouet al\.\([2021](https://arxiv.org/html/2605.06814#bib.bib5)\)reduces model complexity \(vertical shift\) and Data \(D\) distillation reduces data complexityLei and Tao \([2023](https://arxiv.org/html/2605.06814#bib.bib8)\)\(horizontal shift\), M2D distillation trades model complexity for data complexity so that humans can compare modelsℳ\\mathcal\{M\}andmmbased on the differences between the corresponding datasetsDDand𝔻\\mathbb\{D\}\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x1.png)Figure 1:Increasing transparency of a fair Graph Convolutional Network \(GCN\) via M2D using a standard GCN as a comparison\. The mechanism that enables the Fair GCN to generate fair predictions is not transparent to the user \(middle\)\. M2D makes the Fair GCN more transparent by distilling it into an augmented graph so that the GCN predictions are fair \(right\)\. Intuitively, the graph augmentation should capture how the Fair GCN addresses the bias in the features and topology of the input graph\.Figure[1](https://arxiv.org/html/2605.06814#S1.F1)illustrates our approach in the context of fair machine learning, where the goal is to better understand how a fair Graph Convolutional Network \(GCN\) mitigates biases in node classification compared to a simple GCN\. More specifically, although a fair GCN produces bias\-mitigated predictions, the internal mechanisms that enforce fairness are often opaque \(middle\)\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x2.png)Figure 2:Distillation design space with data \(x\-axis\) and model \(y\-axis\) complexity\. Knowledge \(K\) distillation reduces model complexity \(vertical\), and data \(D\) distillation reduces data complexity \(horizontal\)\. Model\-to\-Data \(M2D\) distillation transfers capacity from the model to the data \(diagonal\)\.By shifting complexity to the data \(right\), fairness is no longer hidden within opaque adversarial training or complex loss functions but is instead materialized as modifications to the graph\. This is particularly critical for high\-stakes auditing: an auditor can inspect the specific edges added or features modified to mitigate bias, providing a more intuitive explanation than the internal gradients of a Fair GCN\. Furthermore, we show that M2D can effectively ‘capture’ the global context of Graph Transformers, distilling their long\-range dependencies into local feature augmentations that a simple GCN can process\.

M2D has other applications beyond transparency\. The graph augmentations can be shared as benchmarks for simpler models that can be trained under resource constraints \(e\.g\., on edge devices\)\. Moreover, M2D can be applied to reverse\-engineer a black\-box model using only access to its logits\.

We summarize the contributions of our paper as follows: \(1\) we introduce model\-to\-data \(M2D\) distillation as a general framework to exchange model and data complexity in GNNs; \(2\) we instantiate M2D through an iterative framework using feature and structure learning and multi\-objective optimization; and \(3\) we empirically show how M2D increases the transparency of fair GNNs, Graph Attention Networks, and Graph Transformers\.

### 1\.1Related Work

##### Graph explainers:

GNN explainers focus on instance\-level explanations, aiming to interpret individual predictions\. Gradient\-based methods \(e\.g\., SA, Guided BP, CAM, Grad\-CAM\) attribute predictions using gradients or activations; perturbation\-based methods \(e\.g\., GNNExplainer, PGExplainer, GraphMask, DnX, SubgraphX\) learn masks over nodes, edges, or features to preserve predictions; surrogate methods \(e\.g\., GraphLime, RelEx, PGM\-Explainer\) approximate local model behavior with interpretable models; and decomposition methods \(e\.g\., LRP, Excitation BP, GNN\-LRP\) distribute prediction scores back to the input spaceBaldassarre and Azizpour \([2019](https://arxiv.org/html/2605.06814#bib.bib44)\); Popeet al\.\([2019](https://arxiv.org/html/2605.06814#bib.bib45)\); Yinget al\.\([2019](https://arxiv.org/html/2605.06814#bib.bib46)\); Luoet al\.\([2020](https://arxiv.org/html/2605.06814#bib.bib42)\); Schlichtkrullet al\.\([2020](https://arxiv.org/html/2605.06814#bib.bib48)\); Pereiraet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib43)\); Yuanet al\.\([2021](https://arxiv.org/html/2605.06814#bib.bib47)\); Huanget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib49)\); Zhanget al\.\([2021b](https://arxiv.org/html/2605.06814#bib.bib50)\); Vu and Thai \([2020](https://arxiv.org/html/2605.06814#bib.bib51)\)\. Model\-level explainers \(e\.g\., XGNN, PAGE, GNNInterpreter, GLGExplainer, GCFExplainer, MOSE\) seek to provide global explanations by identifying patterns, subgraphs, or concepts that consistently influence predictions across a target classShinet al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib56)\); Vasilcoiuet al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib57)\); Xuanyuanet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib58)\); Kanget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib59)\); Liuet al\.\([2025](https://arxiv.org/html/2605.06814#bib.bib54)\)\. While these methods explain what drives predictions, they do not explain why a particular architecture outperforms another on a dataset\. In contrast, M2D targets architectural transparency by translating model behavior into explicit modifications of the data, enabling direct comparison of models through their induced data transformations\.

##### Knowledge distillation on graphs:

Knowledge distillation on graphs varies along two axes: the type of knowledge transferred and the distillation framework\. Existing methods distill either logits by aligning output distributions \(e\.g\., TinyGNN, GFKD, DFAD\-GNN, KDGA, CPF, GLNN\)Yanet al\.\([2020](https://arxiv.org/html/2605.06814#bib.bib17)\); Deng and Zhang \([2021](https://arxiv.org/html/2605.06814#bib.bib18)\); Zhuanget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib19)\); Wuet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib20)\); Yanget al\.\([2021](https://arxiv.org/html/2605.06814#bib.bib21)\); Zhanget al\.\([2022b](https://arxiv.org/html/2605.06814#bib.bib22)\), structural knowledge by preserving local and global topology \(e\.g\., LSP, FreeKD, Alignahead, GNN\-SD, CKD, G\-CRD, ROD, MSKD, Cold Brew, GKD, NOSMOG\)Yanget al\.\([2020](https://arxiv.org/html/2605.06814#bib.bib25)\); Fenget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib32)\); Guoet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib27)\); Chenet al\.\([2020b](https://arxiv.org/html/2605.06814#bib.bib33)\); Wanget al\.\([2022a](https://arxiv.org/html/2605.06814#bib.bib34)\); Joshiet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib26)\); Zhanget al\.\([2021a](https://arxiv.org/html/2605.06814#bib.bib29),[2022a](https://arxiv.org/html/2605.06814#bib.bib35)\); Zhenget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib23)\); Yanget al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib28)\); Tianet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib24)\), or intermediate embeddings by matching representations across layers \(e\.g\., T2\-GNN, SAIL, GraphAKD, RDD\)Huoet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib31)\); Yuet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib36)\); Heet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib30)\); Zhanget al\.\([2020](https://arxiv.org/html/2605.06814#bib.bib6)\)\. From a framework perspective, prior work includes teacher\-free self\-distillation \(e\.g\., GNN\-SD, CKD, RDD\) and teacher–student paradigm, where pretrained GNNs transfer knowledge to either compact GNNs \(e\.g\., TinyGNN, GraphAKD, LSP\) or simpler models such as MLPs \(e\.g\., CPF, GLNN, Cold Brew, NOSMOG\)\.

##### Data distillation on graphs:

Graph data distillation is often referred to as graph condensation \(GC\)\. GC methods can be categorized by their objectives and aim to preserve task\-specific performance, enabling GNNs trained on condensed graphs to match those trained on the original graph \(e\.g\., GCond, SFGC, CTRL, OpenGC, GC\-STNK, GCSR, KiDD\)Hashemiet al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib61)\); Liuet al\.\([2023b](https://arxiv.org/html/2605.06814#bib.bib62)\); Zhanget al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib63)\); Gaoet al\.\([2024b](https://arxiv.org/html/2605.06814#bib.bib64)\); Zhenget al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib66)\); Liuet al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib67)\)\. Generalized GC methods, such as GDEM, SDDD, OpenGC, etc\., focus on learning condensed graphs that generalize across different models and tasks by retaining essential structural and feature informationLiuet al\.\([2023a](https://arxiv.org/html/2605.06814#bib.bib68)\); Xuet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib69)\); Gaoet al\.\([2024b](https://arxiv.org/html/2605.06814#bib.bib64)\)\. DosCond, CaT, and EXGC seek to accelerate the condensation process by improving stages such as encoding, optimization, and generationJinet al\.\([2022](https://arxiv.org/html/2605.06814#bib.bib73)\); Gaoet al\.\([2024a](https://arxiv.org/html/2605.06814#bib.bib74)\); Fanget al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib75)\)\. FGD and GCARe incorporate constraints or regularization to mitigate bias and promote equitable representationsFenget al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib70)\); Maoet al\.\([2023](https://arxiv.org/html/2605.06814#bib.bib71)\)\. RobGC aims to filter noise and preserve core, causal information for reliable performance in real\-world settingsGaoet al\.\([2025](https://arxiv.org/html/2605.06814#bib.bib72)\)\.

M2D differs from both knowledge and data distillation by enabling a trade\-off between model and data complexity, allowing a simple model to recover a complex teacher’s behavior via an augmented graph\. This positions M2D in a distinct, underexplored region of the model–data design space\. Our approach differs from traditional graph rewiring, which typically modifies the topology to alleviate message\-passing bottlenecks such as oversmoothing or oversquashingAttaliet al\.\([2024](https://arxiv.org/html/2605.06814#bib.bib77)\)\. While prior work uses iterative structure learning to improve robustnessChenet al\.\([2020a](https://arxiv.org/html/2605.06814#bib.bib37)\), M2D instead iteratively distills teacher behavior into the graph, treating the data as a transparent medium for encoding the teacher’s knowledge\.

## 2Model to Data \(M2D\) Distillation

### 2\.1Problem Definition

We propose Model\-to\-Data Distillation \(M2D\), a new distillation paradigm that transfers complexity from a Graph Neural Network \(GNN\) model to the data\. M2D enriches the data so that a lightweight GNN can achieve performance comparable to a more complex one \(see Figure[2](https://arxiv.org/html/2605.06814#S1.F2)\)\.

Let𝒢=\(𝒱,ℰ,𝐗\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\},\\mathbf\{X\}\)be an undirected graph where𝒱\\mathcal\{V\}is the set of nodes,ℰ⊆𝒱×𝒱\\mathcal\{E\}\\subseteq\\mathcal\{V\}\\times\\mathcal\{V\}is the set of edges, and𝐗∈ℝn×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{n\\times d\}are the node attributes\. The matrix𝐀∈ℝn×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{n\\times n\}is the adjacency matrix of𝒢\\mathcal\{G\}where𝐀u​v=1\\mathbf\{A\}\_\{uv\}=1if there is an edge betweenuuandvvand𝐀u​v=0\\mathbf\{A\}\_\{uv\}=0, otherwise\. LetfTf\_\{T\}be the teacher model with parameterθT\\theta\_\{T\}andfsf\_\{s\}be a student model with parameterθs\\theta\_\{s\}where\|θT\|\>\|θs\|\|\\theta\_\{T\}\|\>\|\\theta\_\{s\}\|\. Moreover, let the prediction of the teacher model and the student model be𝐲^T\\hat\{\\mathbf\{y\}\}\_\{T\}and𝐲^s\\hat\{\\mathbf\{y\}\}\_\{s\}, respectively\.

M2D learns a transformationfgf\_\{g\}with parametersθg\\theta\_\{g\}that enriches the original graph data𝒢\\mathcal\{G\}by generating additional features or modifying the graph structure, yielding𝒢~=\(𝒱,ℰ~,𝐗~\)\\tilde\{\\mathcal\{G\}\}=\(\{\\mathcal\{V\}\},\\tilde\{\\mathcal\{E\}\},\\tilde\{\\mathbf\{X\}\}\)\. We learn the graph transformationfgf\_\{g\}and the student modelfsf\_\{s\}jointly by optimizing the following objective:

minθg,θs⁡ℒdis​\(fT​\(𝒢\),fs​\(𝒢~\)\)\+ℒcls​\(fs​\(𝒢~\),𝐲\)−𝒮​\(𝒢,𝒢~\),\\min\_\{\\theta\_\{g\},\\,\\theta\_\{s\}\}\\;\\mathcal\{L\}\_\{\\mathrm\{dis\}\}\\\!\\left\(f\_\{T\}\(\\mathcal\{G\}\),f\_\{s\}\(\\tilde\{\\mathcal\{G\}\}\)\\right\)\+\\mathcal\{L\}\_\{\\mathrm\{cls\}\}\\\!\\left\(f\_\{s\}\(\\tilde\{\\mathcal\{G\}\}\),\\mathbf\{y\}\\right\)\-\\,\\mathcal\{S\}\\\!\\left\(\\mathcal\{G\},\\tilde\{\\mathcal\{G\}\}\\right\),\(1\)where𝒢~=fg​\(𝒢\)\\tilde\{\\mathcal\{G\}\}=f\_\{g\}\(\\mathcal\{G\}\)is the augmented graph,ℒdis\\mathcal\{L\}\_\{\\mathrm\{dis\}\}is the distance between the outputs of the teacher modelfTf\_\{T\}and student modelfsf\_\{s\},ℒcls\\mathcal\{L\}\_\{\\mathrm\{cls\}\}is the classification loss between true labels𝐲\\mathbf\{y\}and student output, and𝒮\\mathcal\{S\}is a function that computes the similarity between𝒢\\mathcal\{G\}and𝒢~\\tilde\{\\mathcal\{G\}\}\. The term𝒮​\(𝒢,𝒢~\)\\mathcal\{S\}\(\\mathcal\{G\},\\tilde\{\\mathcal\{G\}\}\)enforces that the model knowledge is distilled into the data while introducing only minimal modifications to the original graph\. In general, both the teacher modelfTf\_\{T\}and the student modelfsf\_\{s\}can be arbitrary architectures\. In this work, however, we restrict our focus to GNNs and Graph Transformers \(GTs\) as teacher models and GNNs and MLPs as student models\. As our motivation is to increase the transparency of the teacher model, our framework assumes only partial access to its logits\. Let𝒱tr∈𝒱\\mathcal\{V\}\_\{\\mathrm\{tr\}\}\\in\\mathcal\{V\}denote the set of nodes for which both teacher logits and ground\-truth labels are available\.

### 2\.2Proposed Model

We propose a solution to the M2D optimization problem of Eq[1](https://arxiv.org/html/2605.06814#S2.E1)\. The learning process alternates between \(i\) updating the graph topology and node features using current embeddings and \(ii\) updating embeddings using the augmented graph\. Let the student GNNfsf\_\{s\}generate node embedding𝐇s∈ℝn×dh\\mathbf\{H\}\_\{s\}\\in\\mathbb\{R\}^\{n\\times d\_\{h\}\}\. We decompose the transformationfgf\_\{g\}into two components: a feature Learnerfϕf\_\{\\phi\}that learns new node features and a structure learnerfaf\_\{a\}that learns new adjacency weights\. Figure[3](https://arxiv.org/html/2605.06814#S2.F3)shows the flow diagram of the proposed model\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x3.png)Figure 3:The feature and structure learners iteratively refine features and structure by optimizingℒcls,ℒdis,ℒdiv\\mathcal\{L\}\_\{\\mathrm\{cls\}\},\\mathcal\{L\}\_\{\\mathrm\{dis\}\},\\mathcal\{L\}\_\{\\mathrm\{div\}\}andℒcls,ℒdis,ℒgraph\\mathcal\{L\}\_\{\\mathrm\{cls\}\},\\mathcal\{L\}\_\{\\mathrm\{dis\}\},\\mathcal\{L\}\_\{\\mathrm\{graph\}\}, respectively\. The student model is then trained on the learned graph, optimizingℒcls\\mathcal\{L\}\_\{\\mathrm\{cls\}\}andℒdis\\mathcal\{L\}\_\{\\mathrm\{dis\}\}to align with the teacher while improving predictive performance\.![Refer to caption](https://arxiv.org/html/2605.06814v1/x4.png)Figure 4:Accuracy–fairness trade\-offs for FairGNN, FairVGNN, and NIFTY and their distilled variants\. The y\-axis shows accuracy, while the x\-axis reports 1 \- DP \(first three columns\) and 1 \- EQOP \(last three columns\)\. Points closer to the top\-right indicate better performance\. Results highlight that bias in NBA is primarily feature\-driven, whereas synthetic and German datasets exhibit bias in both features and topology, requiring joint distillation to recover similar fairness\.
### 2\.3Learning new node features and adjacency weights

##### Feature Learner:

Letfϕ:ℝdh→ℝdff\_\{\\phi\}:\\mathbb\{R\}^\{d\_\{h\}\}\\rightarrow\\mathbb\{R\}^\{d\_\{f\}\}be a function that learns new features of dimensiondfd\_\{f\}and constructs new feature matrix𝐗~\\tilde\{\\mathbf\{X\}\}at timettfrom node embeddings𝐇s\\mathbf\{H\}\_\{s\}at timet−1t\-1as follows:

𝐙t=fϕ​\(𝐇st−1\)∈ℝn×df,𝐗~t=\[𝐗∥𝐙t\]∈ℝn×d\+df\.\\mathbf\{Z\}^\{t\}=f\_\{\\phi\}\(\\mathbf\{H\}^\{t\-1\}\_\{s\}\)\\in\\mathbb\{R\}^\{n\\times d\_\{f\}\},\\quad\\tilde\{\\mathbf\{X\}\}^\{t\}=\[\\mathbf\{X\}\\mathbin\{\\\|\}\\mathbf\{Z\}^\{t\}\]\\in\\mathbb\{R\}^\{n\\times d\+d\_\{f\}\}\.The initial node embeddings𝐇s0\\mathbf\{H\}^\{0\}\_\{s\}are initialized asfs​\(𝐗,𝐀\)f\_\{s\}\(\\mathbf\{X\},\\mathbf\{A\}\)\.

##### Structure Learner:

We learn a new graph structure by predicting pairwise node relationships from their embeddings\. Given node representations𝐇st−1\\mathbf\{H\}^\{t\-1\}\_\{s\}, letfa:ℝ2​dh→\[0,1\]f\_\{a\}:\\mathbb\{R\}^\{2d\_\{h\}\}\\rightarrow\[0,1\]be a function that learns a new weighted adjacency matrix\. The learned matrix is defined element\-wise as:

𝐖u​vt\\displaystyle\\mathbf\{W\}^\{t\}\_\{uv\}=fa​\(𝐡s,ut−1,𝐡s,vt−1\),𝐖t∈\[0,1\]n×n,\\displaystyle=f\_\{a\}\\big\(\\mathbf\{h\}^\{t\-1\}\_\{s,u\},\\mathbf\{h\}^\{t\-1\}\_\{s,v\}\\big\),\\quad\\mathbf\{W\}^\{t\}\\in\[0,1\]^\{n\\times n\},where𝐡s,ut−1\\mathbf\{h\}^\{t\-1\}\_\{s,u\}and𝐡s,vt−1\\mathbf\{h\}^\{t\-1\}\_\{s,v\}are embeddings of nodeuuandvvat timet−1t\-1\. For directed graphs,faf\_\{a\}operates on the concatenation of node embeddings:

𝐖u​vt\\displaystyle\\mathbf\{W\}^\{t\}\_\{uv\}=fa​\(\[𝐡s,ut−1∥𝐡s,vt−1\]\)\.\\displaystyle=f\_\{a\}\\big\(\[\\mathbf\{h\}^\{t\-1\}\_\{s,u\}\\,\\\|\\,\\mathbf\{h\}^\{t\-1\}\_\{s,v\}\]\\big\)\.
For undirected graphs, we enforce permutation invariance by symmetrizing the input:

𝐖u​vt\\displaystyle\\mathbf\{W\}^\{t\}\_\{uv\}=fa​\(𝐡s,ut−1\+𝐡s,vt−1∥\|𝐡s,ut−1−𝐡s,vt−1\|\),\\displaystyle=f\_\{a\}\\big\(\\mathbf\{h\}^\{t\-1\}\_\{s,u\}\+\\mathbf\{h\}^\{t\-1\}\_\{s,v\}\\,\\\|\\,~\\lvert\\mathbf\{h\}^\{t\-1\}\_\{s,u\}\-\\mathbf\{h\}^\{t\-1\}\_\{s,v\}\\rvert\\big\),which ensures𝐖u​vt=𝐖v​ut\\mathbf\{W\}^\{t\}\_\{uv\}=\\mathbf\{W\}^\{t\}\_\{vu\}\. The final weighted adjacency matrix at iterationttis a combination of the original graph with trained and untrained weights:

𝐀~u​vt=\(1−γ\)​𝐀u​v\+γ​\(\(1−β\)​𝐖u​vt\+β​s​\(𝐱u,𝐱v\)\),\\displaystyle\\tilde\{\\mathbf\{A\}\}^\{t\}\_\{uv\}=\(1\-\\gamma\)\\mathbf\{A\}\_\{uv\}\+\\gamma\(\(1\-\\beta\)\\mathbf\{W\}^\{t\}\_\{uv\}\+\\beta s\(\\mathbf\{x\}\_\{u\},\\mathbf\{x\}\_\{v\}\)\),whereγ\\gammaandβ\\betaare hyperparameters, ands​\(𝐱u,𝐱v\)s\(\\mathbf\{x\}\_\{u\},\\mathbf\{x\}\_\{v\}\)is the cosine similarity betweenuuandvvbased on original features\. We then use𝐗~t\\tilde\{\\mathbf\{X\}\}^\{t\}and𝐀~t\\tilde\{\\mathbf\{A\}\}^\{t\}to generate representations𝐇st=fs​\(𝐗~t,𝐀~t\)\\mathbf\{H\}^\{t\}\_\{s\}=f\_\{s\}\(\\tilde\{\\mathbf\{X\}\}^\{t\},\\tilde\{\\mathbf\{A\}\}^\{t\}\)at iterationtt\. The final graph is generated either whenℒcls\\mathcal\{L\}\_\{\\mathrm\{cls\}\}converges or after a fixed number of iterations\.

### 2\.4Training Objective

Let𝒢~t=\(𝒱,𝐀~t,𝐗~t\)\\tilde\{\\mathcal\{G\}\}^\{t\}\\\!=\\\!\(\\mathcal\{V\},\{\\tilde\{\\mathbf\{A\}\}\}^\{t\},\\tilde\{\\mathbf\{X\}\}^\{t\}\)be the augmented graph at timett\. The student encoderfsf\_\{s\}produces node representations𝐇st=fs​\(𝐀~t,𝐗~t\)\\mathbf\{H\}^\{t\}\_\{s\}\\\!=\\\!f\_\{s\}\(\{\\tilde\{\\mathbf\{A\}\}\}^\{t\},\\tilde\{\\mathbf\{X\}\}^\{t\}\), which are mapped to class logits𝐲^st=fc​\(𝐇st\)\\hat\{\\mathbf\{y\}\}^\{t\}\_\{s\}\\\!=\\\!f\_\{c\}\(\\mathbf\{H\}^\{t\}\_\{s\}\)using a classifierfcf\_\{c\}\. We partition the model parameters into groupsΘs=\{θs,θc\}\\Theta\_\{s\}\\\!=\\\!\\\{\\theta\_\{s\},\\theta\_\{c\}\\\}andΘg=\{θϕ,θa\}\\Theta\_\{g\}\\\!=\\\!\\\{\\theta\_\{\\phi\},\\theta\_\{a\}\\\}, whereΘs\\Theta\_\{s\}denotes the student encoder and classifier parameters andΘg\\Theta\_\{g\}denotes the graph augmentation parameters\.

##### Classification lossℒcls\\mathcal\{L\}\_\{\\mathrm\{cls\}\}:

We use the standard supervised cross\-entropy loss\.

ℒcls=−1\|𝒱tr\|​∑u∈𝒱tr𝐲u⊤​log⁡𝐲^s,ut\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{cls\}\}=\-\\frac\{1\}\{\|\\mathcal\{V\}\_\{\\mathrm\{tr\}\}\|\}\\sum\_\{u\\in\\mathcal\{V\}\_\{\\mathrm\{tr\}\}\}\\mathbf\{y\}\_\{u\}^\{\\top\}\\log\\hat\{\\mathbf\{y\}\}^\{t\}\_\{s,u\}\.\(2\)

##### Distillation lossℒdis\\mathcal\{L\}\_\{\\mathrm\{dis\}\}:

We align student and teacher predictive distributions using temperature\-scaled soft targets via the Kullback–Leibler \(KL\) Divergence\. Let the teacher’s logits be𝐲^T\\hat\{\\mathbf\{y\}\}\_\{T\}, then:

ℒdis=τ21\|𝒱tr\|∑u∈𝒱trKL\(𝐲^T,uτ\|\|𝐲^s,utτ\)\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{dis\}\}=\\tau^\{2\}\\frac\{1\}\{\|\\mathcal\{V\}\_\{\\mathrm\{tr\}\}\|\}\\sum\_\{u\\in\\mathcal\{V\}\_\{\\mathrm\{tr\}\}\}\\text\{KL\}\\left\(\\frac\{\\hat\{\\mathbf\{y\}\}\_\{T,u\}\}\{\\tau\}\\middle\|\\middle\|\\frac\{\\hat\{\\mathbf\{y\}\}^\{t\}\_\{s,u\}\}\{\\tau\}\\right\)\.\(3\)whereτ\\tausmooths the distributions, exposing inter\-class similarity and teacher uncertainty that are not captured by teacher logitsHintonet al\.\([2015](https://arxiv.org/html/2605.06814#bib.bib15)\)\. The loss is multiplied byτ2\\tau^\{2\}to preserve gradient magnitudes\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x5.png)Figure 5:Learned features colored by class label and sensitive attribute for the German dataset\. Each row corresponds to a fair GNN with points colored by label \(green/red\) and sensitive attribute \(blue/magenta\)\. Learned features exhibit clearer class separation while mixing the sensitive attribute\.
##### Feature diversity regularizationℒdiv\\mathcal\{L\}\_\{\\mathrm\{div\}\}:

To induce new features to capture complementary information and avoid redundancy with existing ones, we penalize pairwise correlations involving the newly generated feature dimensions\. We first normalize each feature dimension across nodes:𝐗t​\(:,j\)=𝐗~t​\(:,j\)‖𝐗~t​\(:,j\)‖2,j=1,…​d\+df\{\\mathbf\{X\}\}^\{t\}\(:,j\)=\\frac\{\\tilde\{\\mathbf\{X\}\}^\{t\}\(:,j\)\}\{\|\|\\tilde\{\\mathbf\{X\}\}^\{t\}\(:,j\)\|\|\_\{2\}\},~j=1,\.\.\.d\+d\_\{f\}\. The feature correlation matrix is then defined as𝐊t=\(𝐗t\)T​𝐗t\\mathbf\{K\}^\{t\}=\(\{\\mathbf\{X\}\}^\{t\}\)^\{T\}\{\\mathbf\{X\}\}^\{t\}\. Thus,𝐊i​jt\\mathbf\{K\}^\{t\}\_\{ij\}measures the cosine similarity between feature dimensionsiiandjj\. To regularize only the generated features, we define a binary mask𝐌∈\{0,1\}\(d\+df\)×\(d\+df\)\\mathbf\{M\}\\in\\\{0,1\\\}^\{\(d\+d\_\{f\}\)\\times\(d\+d\_\{f\}\)\}such that𝐌i​j=𝟏​\[i\>d∨j\>d\]⋅𝟏​\[i≠j\]\\mathbf\{M\}\_\{ij\}=\\mathbf\{1\}\\left\[i\>d\\;\\vee\\;j\>d\\right\]\\cdot\\mathbf\{1\}\[i\\neq j\]\. Therefore,𝐌i​j=1\\mathbf\{M\}\_\{ij\}=1whenever at least one of the two dimensions corresponds to a generated feature, excluding diagonal self\-correlations\. The diversity regularization loss is computed as follows:

ℒdiv=λdiv​‖𝐌⊙𝐊t‖F2\(d\+df\)2\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{div\}\}=\\lambda\_\{\\mathrm\{div\}\}\\frac\{\\\|\\mathbf\{M\}\\odot\\mathbf\{K\}^\{t\}\\\|\_\{F\}^\{2\}\}\{\(d\+d\_\{f\}\)^\{2\}\}\.\(4\)where⊙\\odotis the element\-wise product andλdiv\\lambda\_\{\\mathrm\{div\}\}is a hyperparameter\.

##### Graph regularization loss,ℒgraph\\mathcal\{L\}\_\{\\mathrm\{graph\}\}:

FollowingKalofolias \([2016](https://arxiv.org/html/2605.06814#bib.bib16)\), we impose additional constraints on𝐖t\\mathbf\{W\}^\{t\}through our graph structure lossℒgraph\\mathcal\{L\}\_\{\\mathrm\{graph\}\}:

ℒgraph=−λdegn​𝟏T​log⁡\(𝐖t​𝟏\)\+λsparsen2​‖𝐖t‖F2\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{graph\}\}=\-\\frac\{\\lambda\_\{\\mathrm\{deg\}\}\}\{n\}\\mathbf\{1\}^\{T\}\\log\(\\mathbf\{W\}^\{t\}\\mathbf\{1\}\)\+\\frac\{\\lambda\_\{\\mathrm\{sparse\}\}\}\{n^\{2\}\}\|\|\\mathbf\{W\}^\{t\}\|\|^\{2\}\_\{F\}\.\(5\)where the first term penalizes zero\-degree nodes and the second term enforces sparsity on the learned adjacency, encouraging the augmented graph to remain close to the original topology\.

The structural and diversity regularization terms are optimized only with respect to the graph augmentation parametersΘg\\Theta\_\{g\}\. Therefore, their gradients with respect to the parametersΘs\\Theta\_\{s\}are constrained to vanish:ΔΘs​ℒdiv=0\\Delta\_\{\\Theta\_\{s\}\}\\mathcal\{L\}\_\{\\mathrm\{div\}\}=0andΔΘs​ℒgraph=0\\Delta\_\{\\Theta\_\{s\}\}\\mathcal\{L\}\_\{\\mathrm\{graph\}\}=0\. At iterationtt, the student parametersΘs\\Theta\_\{s\}are optimized using the supervised objective comprising the classification and distillation losses, whereas the graph modifier parametersΘg\\Theta\_\{g\}are optimized using both the supervised objective and the structural regularization terms, including the graph and feature diversity losses:

Θs∗\\displaystyle\\Theta\_\{s\}^\{\*\}=argminΘs\(1−λ\)disℒcls\+λdisℒdis\.\\displaystyle=\\arg\\min\_\{\\Theta\_\{s\}\}~\(1\-\\lambda\{\{\}\_\{\\mathrm\{dis\}\}\}\)\\mathcal\{L\}\_\{\\mathrm\{cls\}\}\+\\lambda\_\{\\mathrm\{dis\}\}\\mathcal\{L\}\_\{\\mathrm\{dis\}\}\.Θg∗\\displaystyle\\Theta\_\{g\}^\{\*\}=argminΘg\(1−λ\)disℒcls\+λdisℒdis\+ℒdiv\+ℒgraph\.\\displaystyle=\\arg\\min\_\{\\Theta\_\{g\}\}~\(1\-\\lambda\{\{\}\_\{\\mathrm\{dis\}\}\}\)\\mathcal\{L\}\_\{\\mathrm\{cls\}\}\+\\lambda\_\{\\mathrm\{dis\}\}\\mathcal\{L\}\_\{\\mathrm\{dis\}\}\+\\mathcal\{L\}\_\{\\mathrm\{div\}\}\+\\mathcal\{L\}\_\{\\mathrm\{graph\}\}\.whereλdis\\lambda\_\{\\mathrm\{dis\}\}is a hyperparameter that regulates the importance of the distillation loss\.

## 3Theoretical Analysis of M2D Transparency

While structural graph modifications are relatively easy to audit, continuous synthetic features are often less transparent\. To better understand the learned features, we present Theorems[1](https://arxiv.org/html/2605.06814#Thmtheorem1),[2](https://arxiv.org/html/2605.06814#Thmtheorem2), and Corollary[1](https://arxiv.org/html/2605.06814#Thmcorollary1)\. Theorem[1](https://arxiv.org/html/2605.06814#Thmtheorem1)shows that fairness in the teacher representations can be transferred to the distilled features\. Theorem[2](https://arxiv.org/html/2605.06814#Thmtheorem2)proves that the M2D learns features that encode the teacher’s attention\-weighted neighborhood aggregation, while Corollary[1](https://arxiv.org/html/2605.06814#Thmcorollary1)shows that higher teacher attention leads to higher similarity in the distilled feature space\. We focus on feature augmentations because they are harder to audit than structural modifications, though similar analyses can also be derived for learned structure\. All proofs are provided in the Appendix\.

###### Theorem 1\.

Assume teacher representations𝐇T\\mathbf\{H\}\_\{T\}satisfy demographic parity with respect to the sensitive attribute𝐬\\mathbf\{s\}, i\.e\.,𝐇T⟂𝐬\\mathbf\{H\}\_\{T\}\\perp\\mathbf\{s\},‖𝐲^s−𝐲^T‖22\\\|\\hat\{\\mathbf\{y\}\}\_\{s\}\-\\hat\{\\mathbf\{y\}\}\_\{T\}\\\|\_\{2\}^\{2\}≈0\\approx 0, linear teacher and student prediction heads, and the feature mapϕ:ℋs→𝒵\\phi:\\mathcal\{H\}\_\{s\}\\to\\mathcal\{Z\}is a Borel measurable function\. Then M2D distillation can learn features𝐙=ϕ​\(𝐇s\)\\mathbf\{Z\}=\\phi\(\\mathbf\{H\}\_\{s\}\), such that𝐙⟂𝐬\\mathbf\{Z\}\\perp\\mathbf\{s\}\.

###### Theorem 2\.

Let the teacher be a GAT or GT with attention coefficientsαi​j\\alpha\_\{ij\}\. Assume linear teacher and student prediction heads, and sufficiently small distillation error\. Then, for each nodeii, there exists a learned feature vector𝐳i\\mathbf\{z\}\_\{i\}such that𝐳i=fϕ​\(∑j∈𝒩​\(i\)αi​j​𝐌𝐱j\+𝐫i\+𝐧i\),\\mathbf\{z\}\_\{i\}=f\_\{\\phi\}\\\!\\left\(\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{r\}\_\{i\}\+\\mathbf\{n\}\_\{i\}\\right\),where𝐌\\mathbf\{M\}is a linear transformation,𝐫i\\mathbf\{r\}\_\{i\}captures the residual term of the linear approximation of the teacher and the distillation error, and𝐧i∈Null​\(𝐔s\)\\mathbf\{n\}\_\{i\}\\in\\mathrm\{Null\}\(\\mathbf\{U\}\_\{s\}\)lies in the null space of the student prediction head\.

###### Corollary 1\.

Under the conditions of Theorem[2](https://arxiv.org/html/2605.06814#Thmtheorem2), let𝐇T\\mathbf\{H\}\_\{T\}be teacher representations from a GAT or GT with attention coefficientsα\\alpha\. Assume a homophilic graph and a monotone functionψ\\psisuch thatsim​\(𝐡T,i,𝐡T,j\)=ψ​\(ei​j\)\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,j\}\)=\\psi\(e\_\{ij\}\), whereei​je\_\{ij\}are pre\-softmax attention scores\. Then, for anyj,k∈𝒩​\(i\)j,k\\in\\mathcal\{N\}\(i\),αi​j\>αi​k\\alpha\_\{ij\}\>\\alpha\_\{ik\}impliessim​\(𝐡T,i,𝐡T,j\)\>sim​\(𝐡T,i,𝐡T,k\)\.\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,j\}\)\>\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,k\}\)\.Further, if𝐫i≈0\\mathbf\{r\}\_\{i\}\\approx 0andfϕf\_\{\\phi\}is order\-preserving with respect tosim​\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\), thensim​\(𝐳i,𝐳j\)\>sim​\(𝐳i,𝐳k\)\.\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)\>\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{k\}\)\.

## 4Experiments and Results

We evaluate how M2D enhances GNN transparency, focusing on fair GNNs, Graph Attention Networks \(GAT\), and Graph Transformers \(GT\)\. Our goal is to analyze how model complexity and fairness mechanisms transfer into the data space\.

### 4\.1Experimental Setup

We consider three variants of M2D: feature\-only \(feat\), adjacency\-only \(adj\), and joint feature–adjacency \(both\)\. We additionally evaluate a baseline where the teacher is distilled into the student without M2D \(w/o\)\. Both the feature learnerfϕf\_\{\\phi\}and the adjacency learnerfaf\_\{a\}are implemented as MLPs\. Additional details about the experimental setup are provided in the Appendix\.

### 4\.2Fair GNN Analysis

We apply M2D to the fairness mechanisms of three fair GNN architectures, FairGNN, FairVGNN, and NIFTYDai and Wang \([2021](https://arxiv.org/html/2605.06814#bib.bib78)\); Wanget al\.\([2022b](https://arxiv.org/html/2605.06814#bib.bib79)\); Agarwalet al\.\([2021](https://arxiv.org/html/2605.06814#bib.bib80)\)as teacher models, with a GCN as the student\. We evaluate utility via accuracy and fairness using Demographic Parity \(DP\) and Equal Opportunity \(EQOP\)\. Details on datasets, baselines, hyperparameters, and evaluation protocols are provided in the Appendix\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x6.png)Figure 6:Original and augmented structure for a sample of 40 nodes from the German dataset, highlighting edges that are modified for fair prediction\. Edge weights are encoded by color and width\. Node colors denote class labels, and shapes represent sensitive attributes\.Figure[4](https://arxiv.org/html/2605.06814#S2.F4)shows the accuracy–fairness trade\-off for each model and its distilled variants\. The y\-axis reports accuracy, while the x\-axis shows1−DP1\\\!\-\\\!\\text\{DP\}\(first three columns\) and1−EQOP1\\\!\-\\\!\\text\{EQOP\}\(last three\)\. Points closer to the top\-right indicate a better trade\-off\. In theNBAdataset, bias is mainly feature\-driven, as fairness improves with feature\-only distillation\. In contrast, thesimulationand theGermandatasets exhibit bias in both features and structure, requiring joint feature–adjacency distillation\.

We further analyze the learned features and structure using GCN \(Both\) on the German dataset\. For each teacher, we learn two features and visualize them in Figure[5](https://arxiv.org/html/2605.06814#S2.F5)\. The features exhibit clear class separation while mixing sensitive groups\. Figure[6](https://arxiv.org/html/2605.06814#S4.F6)shows how structure learning augments a 40\-node subgraph from the same dataset\. Compared to the original graph \(left\), the learned graph \(right\) shows removed or weakened biased connections and strengthened or added unbiased ones\. For example, an edge between a male positive node and a male negative node is removed to prevent the male negative node from being biased toward a positive prediction\.

### 4\.3GAT and GT Analysis

More complex architectures, such as GAT and Graph Transformers, are often preferred over simpler models like GCN due to their superior performance\. We analyze the advantage of GAT and GraphormerYinget al\.\([2021](https://arxiv.org/html/2605.06814#bib.bib81)\), a GT, over GCN and MLP \(see Appendix\) using our approach\. Table[1](https://arxiv.org/html/2605.06814#S4.T1)reports node classification accuracy across multiple datasets, comparing teacher models \(GAT and GT\) against the GCN student model under different distillation variants\.

Table 1:Node classification accuracy comparing teacher models \(GAT, GT\) against student GCN under different settings\. ‘Van’ is the student without any distillation, ‘w/o’ is the standard knowledge distillation \(without M2D\), and ‘Feat\.’, ‘Adj’ and ‘Both’ are M2D applied to features, adjacency, and both, respectively\. Results show that M2D, particularly Feat\. and Both consistently improve student performance and can match or surpass teacher accuracy across datasets\.M2D consistently improves student performance compared to both a vanilla student \(without distillation\) and standard distillation\. Feature\-based distillation provides the most reliable gains, while adjacency\-only distillation is less effective\. Joint distillation \(both\) typically achieves the second\-best results, in some cases matching or even surpassing the teacher models\. These findings demonstrate that M2D effectively transfers knowledge from more complex architectures, such as GAT and GT, to simpler models\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x7.png)Figure 7:Analysis of learned node features with respect to the attention of the teacher GAT\. For each dataset and feature type, we partition the range of feature similaritysim​\(𝐳i,𝐳j\)\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)into bins and compute the average attention weight within each bin for existing edges\. The increasing trend in homophilic datasets \(Cora, Citeseer, Photo\) indicates that higher attention weights correspond to higher similarity in the learned features, showing that the features capture the attention mechanism of the GAT teacher \(Corollary[1](https://arxiv.org/html/2605.06814#Thmcorollary1)\)\. We can also see the trend in heterophilic datasets \(Cornell, Texas\)\.Figure[7](https://arxiv.org/html/2605.06814#S4.F7)analyzes how the learned node features align with the attention mechanism of a teacher GAT model\. We first compute pairwise feature similarity using RBF Kernel \(σ=1\\sigma=1\), i\.e\.,sim​\(zi,zj\)\\mathrm\{sim\}\(z\_\{i\},z\_\{j\}\), and group node pairs into bins based on their similarity values\. For each bin, we measure the average attention weight of the teacher’s existing edges\. For homophilic datasets \(Cora, Citeseer, Photo\), we observe a clear increasing trend: higher similarity corresponds to larger attention weights\. In heterophilic datasets \(Cornell, Texas\), a similar positive relationship is still observed; however, it is weaker and more dispersed, indicating greater variability in how attention aligns with feature similarity\. This difference highlights how the alignment between feature similarity and attention depends on the underlying graph structure\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x8.png)Figure 8:Visualization of edge weights on Cora \(homophilic\) using row\-normalized M2D learned weights \(left\) and layer\-1 GAT attention coefficients \(right\)\. Edge color intensity and line width are proportional to the magnitude of the corresponding weight \(darker and thicker edges indicate higher importance\)\. Learned weights are normalized per node, aligning them with the probabilistic interpretation of attention coefficients\. Both visualizations use an identical layout for direct comparison\. The learned weights approximate the GAT coefficients \(Pearson’s correlation =0\.840\.84\)\.Figure[8](https://arxiv.org/html/2605.06814#S4.F8)visualizes edge importance on Cora \(homophilic\) using row\-normalized M2D edge weights and GAT attention coefficients \(Layer 1\)\. Edge color and line width reflect the magnitude of importance \(darker and thicker edges indicating higher values\)\. The M2D weights are normalized per node \(rows sum to one\), enabling a direct probabilistic comparison with attention coefficients\. Using an identical layout, both visualizations exhibit highly similar patterns, indicating that the learned edge weights closely match the GAT attention mechanism\.

## 5Conclusion and Future Work

We have introduced Model\-to\-Data \(M2D\) distillation, motivated by the need for increasing the transparency of Graph Neural Networks\. M2D transfers complexity from a complex model to the data \(graph structure and features\), enabling a simpler student model to achieve similar performance as the complex teacher\. Our results show that M2D helps us to better understand how fair GNNs mitigate biases in the data and why complex GNNs, such as Graph Attention Networks and Graph Transformers, outperform Graph Convolution Networks on some benchmarks\.

Limitations and future work\.We apply M2D to relatively small datasets, as scaling up our augmentation approach is a challenge, especially for structure learning, which we leave as future work\. In practice, one can also apply M2D to a smaller sample from a larger graph\. Moreover, we want to extend our approach to consider additional augmentations \(e\.g\., feature deletion, node additions\)\.

## References

- \[1\]\(2021\)Towards a unified framework for fair and stable graph representation learning\.InUncertainty in Artificial Intelligence,pp\. 2114–2124\.Cited by:[Baselines](https://arxiv.org/html/2605.06814#Ax2.SSx2.p4.1),[§4\.2](https://arxiv.org/html/2605.06814#S4.SS2.p1.1)\.
- \[2\]H\. Attali, D\. Buscaldi, and N\. Pernelle\(2024\)Rewiring techniques to mitigate oversquashing and oversmoothing in gnns: a survey\.arXiv preprint arXiv:2411\.17429\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p2.1)\.
- \[3\]F\. Baldassarre and H\. Azizpour\(2019\)Explainability techniques for graph convolutional networks\.arXiv preprint arXiv:1905\.13686\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[4\]Y\. Chen, L\. Wu, and M\. Zaki\(2020\)Iterative deep graph learning for graph neural networks: better and robust node embeddings\.Advances in Neural Information Processing Systems33,pp\. 19314–19326\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p2.1)\.
- \[5\]Y\. Chen, Y\. Bian, X\. Xiao, Y\. Rong, T\. Xu, and J\. Huang\(2020\)On self\-distilling graph neural network\.arXiv preprint arXiv:2011\.02255\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[6\]E\. Dai and S\. Wang\(2021\)Say no to the discrimination: learning fair graph neural networks with limited sensitive attribute information\.InProceedings of the 14th ACM International Conference on Web Search and Data Mining,pp\. 680–688\.Cited by:[Dataset for Fairness Evaluation:](https://arxiv.org/html/2605.06814#Ax2.SSx1.SSSx1.p3.1),[Baselines](https://arxiv.org/html/2605.06814#Ax2.SSx2.p2.1),[§4\.2](https://arxiv.org/html/2605.06814#S4.SS2.p1.1)\.
- \[7\]X\. Deng and Z\. Zhang\(2021\)Graph\-free knowledge distillation for graph neural networks\.arXiv preprint arXiv:2105\.07519\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[8\]J\. Fang, X\. Li, Y\. Sui, Y\. Gao, G\. Zhang, K\. Wang, X\. Wang, and X\. He\(2024\)EXGC: bridging efficiency and explainability in graph condensation\.InProceedings of the ACM Web Conference 2024,WWW ’24,New York, NY, USA,pp\. 721–732\.External Links:ISBN 9798400701719,[Link](https://doi.org/10.1145/3589334.3645551),[Document](https://dx.doi.org/10.1145/3589334.3645551)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[9\]K\. Feng, C\. Li, Y\. Yuan, and G\. Wang\(2022\)Freekd: free\-direction knowledge distillation for graph neural networks\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 357–366\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[10\]Q\. Feng, Z\. S\. Jiang, R\. Li, Y\. Wang, N\. Zou, J\. Bian, and X\. Hu\(2023\)Fair graph distillation\.Advances in Neural Information Processing Systems36,pp\. 80644–80660\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[11\]X\. Gao, T\. Chen, Y\. Zang, W\. Zhang, Q\. V\. Hung Nguyen, K\. Zheng, and H\. Yin\(2024\)Graph condensation for inductive node representation learning\.In2024 IEEE 40th International Conference on Data Engineering \(ICDE\),Vol\.,pp\. 3056–3069\.External Links:[Document](https://dx.doi.org/10.1109/ICDE60146.2024.00237)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[12\]X\. Gao, T\. Chen, W\. Zhang, Y\. Li, X\. Sun, and H\. Yin\(2024\)Graph condensation for open\-world graph learning\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 851–862\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[13\]X\. Gao, H\. Yin, T\. Chen, G\. Ye, W\. Zhang, and B\. Cui\(2025\)RobGC: towards robust graph condensation\.IEEE Transactions on Knowledge and Data Engineering37\(8\),pp\. 4791–4804\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2025.3569629)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[14\]A\. P\. García\-Plaza, V\. Fresno, R\. M\. Unanue, and A\. Zubiaga\(2016\)Using fuzzy logic to leverage html markup for web page representation\.IEEE Transactions on Fuzzy Systems25\(4\),pp\. 919–933\.Cited by:[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2),[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2.4.2)\.
- \[15\]J\. Gilmer, S\. S\. Schoenholz, P\. F\. Riley, O\. Vinyals, and G\. E\. Dahl\(2017\)Neural message passing for quantum chemistry\.InInternational Conference on Machine Learning,pp\. 1263–1272\.Cited by:[§1](https://arxiv.org/html/2605.06814#S1.p1.1)\.
- \[16\]J\. Gou, B\. Yu, S\. J\. Maybank, and D\. Tao\(2021\)Knowledge distillation: a survey\.International Journal of Computer Vision129\(6\),pp\. 1789–1819\.Cited by:[§1](https://arxiv.org/html/2605.06814#S1.p3.7)\.
- \[17\]S\. Gunasekar, B\. E\. Woodworth, S\. Bhojanapalli, B\. Neyshabur, and N\. Srebro\(2017\)Implicit regularization in matrix factorization\.Advances in neural information processing systems30\.Cited by:[Proof\.](https://arxiv.org/html/2605.06814#Ax2.SSx4.1.p1.6),[Proof\.](https://arxiv.org/html/2605.06814#Ax2.SSx4.SSSx2.1.p1.7)\.
- \[18\]J\. Guo, D\. Chen, and C\. Wang\(2022\)Alignahead: online cross\-layer knowledge extraction on graph neural networks\.In2022 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[19\]W\. Hamilton, Z\. Ying, and J\. Leskovec\(2017\)Inductive representation learning on large graphs\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2605.06814#S1.p1.1)\.
- \[20\]M\. Hashemi, S\. Gong, J\. Ni, W\. Fan, B\. A\. Prakash, and W\. Jin\(2024\)A comprehensive survey on graph reduction: sparsification, coarsening, and condensation\.arXiv preprint arXiv:2402\.03358\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[21\]H\. He, J\. Wang, Z\. Zhang, and F\. Wu\(2022\)Compressing deep graph neural networks via adversarial knowledge distillation\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 534–544\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[22\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§2\.4](https://arxiv.org/html/2605.06814#S2.SS4.SSS0.Px2.p1.3)\.
- \[23\]Hofmann,Hans\(1994\)Statlog \(German Credit Data\)\.Note:UCI Machine Learning RepositoryCited by:[Dataset for Fairness Evaluation:](https://arxiv.org/html/2605.06814#Ax2.SSx1.SSSx1.p2.1)\.
- \[24\]Q\. Huang, M\. Yamada, Y\. Tian, D\. Singh, and Y\. Chang\(2022\)Graphlime: local interpretable model explanations for graph neural networks\.IEEE Transactions on Knowledge and Data Engineering35\(7\),pp\. 6968–6972\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[25\]C\. Huo, D\. Jin, Y\. Li, D\. He, Y\. Yang, and L\. Wu\(2023\)T2\-gnn: graph neural networks for graphs with incomplete features and structure via teacher\-student distillation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 4339–4346\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[26\]W\. Jin, X\. Tang, H\. Jiang, Z\. Li, D\. Zhang, J\. Tang, and B\. Yin\(2022\)Condensing graphs via one\-step gradient matching\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,KDD ’22,New York, NY, USA,pp\. 720–730\.External Links:ISBN 9781450393850,[Link](https://doi.org/10.1145/3534678.3539429),[Document](https://dx.doi.org/10.1145/3534678.3539429)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[27\]C\. K\. Joshi, F\. Liu, X\. Xun, J\. Lin, and C\. S\. Foo\(2022\)On representation knowledge distillation for graph neural networks\.IEEE transactions on Neural Networks and Learning Systems35\(4\),pp\. 4656–4667\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[28\]V\. Kalofolias\(2016\)How to learn a graph from smooth signals\.InArtificial Intelligence and Statistics,pp\. 920–929\.Cited by:[§2\.4](https://arxiv.org/html/2605.06814#S2.SS4.SSS0.Px4.p1.2)\.
- \[29\]X\. Kang, D\. Liang, and Q\. Li\(2022\)GANExplainer: explainability method for graph neural network with generative adversarial nets\.InInternational Conference on Computing and Pattern Recognition,pp\. 297–302\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[30\]T\. N\. Kipf and M\. Welling\(2017\)Semi\-supervised classification with graph convolutional networks\.InICLR,Cited by:[Graph Neural Networks](https://arxiv.org/html/2605.06814#Ax1.p1.4),[§1](https://arxiv.org/html/2605.06814#S1.p1.1)\.
- \[31\]S\. Lei and D\. Tao\(2023\)A comprehensive survey of dataset distillation\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(1\),pp\. 17–32\.Cited by:[§1](https://arxiv.org/html/2605.06814#S1.p3.7)\.
- \[32\]Z\. C\. Lipton\(2018\)The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery\.\.Queue16\(3\),pp\. 31–57\.Cited by:[§1](https://arxiv.org/html/2605.06814#S1.p1.1)\.
- \[33\]L\. Liu, P\. Wan, F\. Zhang, Y\. Zhang, Q\. Liu, and G\. Wang\(2025\)Learning model\-level explanations of graph neural networks via subgraph order embedding space\.Neural Networks191,pp\. 107815\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[34\]Y\. Liu, D\. Bo, and C\. Shi\(2023\)Graph distillation with eigenbasis matching\.arXiv preprint arXiv:2310\.09202\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[35\]Y\. Liu, R\. Qiu, and Z\. Huang\(2023\)Cat: balanced continual graph learning with graph condensation\.In2023 IEEE International Conference on Data Mining \(ICDM\),pp\. 1157–1162\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[36\]Z\. Liu, C\. Zeng, and G\. Zheng\(2024\)Graph data condensation via self\-expressive graph structure reconstruction\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1992–2002\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[37\]D\. Luo, W\. Cheng, D\. Xu, W\. Yu, B\. Zong, H\. Chen, and X\. Zhang\(2020\)Parameterized explainer for graph neural network\.Advances in Neural Information Processing Systems33,pp\. 19620–19631\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[38\]R\. Mao, W\. Fan, and Q\. Li\(2023\)GCARe: mitigating subgroup unfairness in graph condensation through adversarial regularization\.Applied Sciences\.External Links:[Link](https://api.semanticscholar.org/CorpusID:260847452)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[39\]T\. Pereira, E\. Nascimento, L\. E\. Resck, D\. Mesquita, and A\. Souza\(2023\)Distill n’explain: explaining graph neural networks using simple surrogates\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 6199–6214\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[40\]P\. E\. Pope, S\. Kolouri, M\. Rostami, C\. E\. Martin, and H\. Hoffmann\(2019\)Explainability methods for graph convolutional neural networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10772–10781\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[41\]M\. S\. Schlichtkrull, N\. De Cao, and I\. Titov\(2020\)Interpreting graph neural networks for nlp with differentiable edge masking\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[42\]P\. Sen, G\. Namata, M\. Bilgic, L\. Getoor, B\. Galligher, and T\. Eliassi\-Rad\(2008\)Collective classification in network data\.AI Magazine29\(3\),pp\. 93–93\.Cited by:[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2),[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2.4.2)\.
- \[43\]O\. Shchur, M\. Mumme, A\. Bojchevski, and S\. Günnemann\(2018\)Pitfalls of graph neural network evaluation\.arXiv preprint arXiv:1811\.05868\.Cited by:[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2),[Table 2](https://arxiv.org/html/2605.06814#Ax2.T2.4.2)\.
- \[44\]Y\. Shin, S\. Kim, and W\. Shin\(2024\)PAGE: prototype\-based model\-level explanations for graph neural networks\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(10\),pp\. 6559–6576\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[45\]Y\. Tian, C\. Zhang, Z\. Guo, X\. Zhang, and N\. Chawla\(2023\)Learning MLPs on graphs: a unified view of effectiveness, robustness, and efficiency\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Cs3r5KLdoj)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[46\]A\. Vasilcoiu, T\. Stessen, T\. Kersten, and B\. Helvacioglu\(2024\)\[Re\] gnninterpreter: a probabilistic generative model\-level explanation for graph neural networks\.Transactions on Machine Learning Research\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[47\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[Graph Neural Networks](https://arxiv.org/html/2605.06814#Ax1.p5.1)\.
- \[48\]P\. Veličković, G\. Cucurull, A\. Casanova, A\. Romero, P\. Lio, Y\. Bengio,et al\.\(2018\)Graph attention networks\.InInternational Conference on Learning Representations,Cited by:[Graph Neural Networks](https://arxiv.org/html/2605.06814#Ax1.p1.4),[§1](https://arxiv.org/html/2605.06814#S1.p1.1),[§1](https://arxiv.org/html/2605.06814#S1.p2.1)\.
- \[49\]M\. Vu and M\. T\. Thai\(2020\)Pgm\-explainer: probabilistic graphical model explanations for graph neural networks\.Advances in Neural Information Processing Systems33,pp\. 12225–12235\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[50\]C\. Wang, S\. Zhou, K\. Yu, D\. Chen, B\. Li, Y\. Feng, and C\. Chen\(2022\)Collaborative knowledge distillation for heterogeneous information network embedding\.InProceedings of the ACM Web Conference,pp\. 1631–1639\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[51\]Y\. Wang, Y\. Zhao, Y\. Dong, H\. Chen, J\. Li, and T\. Derr\(2022\)Improving fairness in graph neural networks via mitigating sensitive attribute leakage\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1938–1948\.Cited by:[Baselines](https://arxiv.org/html/2605.06814#Ax2.SSx2.p3.1),[§4\.2](https://arxiv.org/html/2605.06814#S4.SS2.p1.1)\.
- \[52\]L\. Wu, H\. Lin, Y\. Huang, and S\. Z\. Li\(2022\)Knowledge distillation improves graph structure augmentation for graph neural networks\.Advances in Neural Information Processing Systems35,pp\. 11815–11827\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[53\]Z\. Xu, Y\. Chen, M\. Pan, H\. Chen, M\. Das, H\. Yang, and H\. Tong\(2023\)Kernel ridge regression\-based graph dataset distillation\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 2850–2861\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[54\]H\. Xuanyuan, P\. Barbiero, D\. Georgiev, L\. C\. Magister, and P\. Liò\(2023\)Global concept\-based interpretability for graph neural networks via neuron analysis\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 10675–10683\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[55\]B\. Yan, C\. Wang, G\. Guo, and Y\. Lou\(2020\)Tinygnn: learning efficient graph neural networks\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 1848–1856\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[56\]C\. Yang, J\. Liu, and C\. Shi\(2021\)Extract the knowledge of graph neural networks and go beyond it: an effective knowledge distillation framework\.InProceedings of the Web Conference,pp\. 1227–1237\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[57\]C\. Yang, Q\. Wu, and J\. Yan\(2022\)Geometric knowledge distillation: topology compression for graph neural networks\.Advances in Neural Information Processing Systems35,pp\. 29761–29775\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[58\]Y\. Yang, J\. Qiu, M\. Song, D\. Tao, and X\. Wang\(2020\)Distilling knowledge from graph convolutional networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 7074–7083\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[59\]C\. Ying, T\. Cai, S\. Luo, S\. Zheng, G\. Ke, D\. He, Y\. Shen, and T\. Liu\(2021\)Do transformers really perform badly for graph representation?\.Advances in Neural Information Processing Systems34,pp\. 28877–28888\.Cited by:[Graph Neural Networks](https://arxiv.org/html/2605.06814#Ax1.p5.1),[Baselines](https://arxiv.org/html/2605.06814#Ax2.SSx2.p5.1),[Experimental Setup](https://arxiv.org/html/2605.06814#Ax3.SSx1.p1.2),[§4\.3](https://arxiv.org/html/2605.06814#S4.SS3.p1.1)\.
- \[60\]Z\. Ying, D\. Bourgeois, J\. You, M\. Zitnik, and J\. Leskovec\(2019\)Gnnexplainer: generating explanations for graph neural networks\.Advances in Neural Information Processing Systems32\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[61\]L\. Yu, S\. Pei, L\. Ding, J\. Zhou, L\. Li, C\. Zhang, and X\. Zhang\(2022\)Sail: self\-augmented graph contrastive learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 8927–8935\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[62\]H\. Yuan, H\. Yu, J\. Wang, K\. Li, and S\. Ji\(2021\)On explainability of graph neural networks via subgraph explorations\.InInternational Conference on Machine Learning,pp\. 12241–12252\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[63\]C\. Zhang, J\. Liu, K\. Dang, and W\. Zhang\(2022\)Multi\-scale distillation from multiple graph neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 4337–4344\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[64\]S\. Zhang, Y\. Liu, Y\. Sun, and N\. Shah\(2022\)Graph\-less neural networks: teaching old mlps new tricks via distillation\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2110.08727)Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[65\]T\. Zhang, Y\. Zhang, K\. Wang, K\. Wang, B\. Yang, K\. Zhang, W\. Shao, P\. Liu, J\. T\. Zhou, and Y\. You\(2024\)Two trades is not baffled: condensing graph via crafting rational gradient matching\.arXiv preprint arXiv:2402\.04924\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[66\]W\. Zhang, Y\. Jiang, Y\. Li, Z\. Sheng, Y\. Shen, X\. Miao, L\. Wang, Z\. Yang, and B\. Cui\(2021\)ROD: reception\-aware online distillation for sparse graphs\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,pp\. 2232–2242\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[67\]W\. Zhang, X\. Miao, Y\. Shao, J\. Jiang, L\. Chen, O\. Ruas, and B\. Cui\(2020\)Reliable data distillation on graph convolutional network\.InProceedings of the 2020 ACM SIGMOD International Conference on Management of Data,pp\. 1399–1414\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[68\]Y\. Zhang, D\. Defazio, and A\. Ramesh\(2021\)Relex: a model\-agnostic relational model explainer\.InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 1042–1049\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px1.p1.1)\.
- \[69\]W\. Zheng, E\. W\. Huang, N\. Rao, S\. Katariya, Z\. Wang, and K\. Subbian\(2022\)Cold brew: distilling graph node representations with incomplete or missing neighborhoods\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.
- \[70\]X\. Zheng, M\. Zhang, C\. Chen, Q\. V\. H\. Nguyen, X\. Zhu, and S\. Pan\(2023\)Structure\-free graph condensation: from large\-scale graphs to condensed graph\-free data\.Advances in Neural Information Processing Systems36,pp\. 6026–6047\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px3.p1.1)\.
- \[71\]Y\. Zhuang, L\. Lyu, C\. Shi, C\. Yang, and L\. Sun\(2022\)Data\-free adversarial knowledge distillation for graph neural networks\.arXiv preprint arXiv:2205\.03811\.Cited by:[§1\.1](https://arxiv.org/html/2605.06814#S1.SS1.SSS0.Px2.p1.1)\.

## Appendix ATechnical appendices and supplementary material

## Graph Neural Networks

We adopt aGraph Convolutional Network \(GCN\)as the student and aGraph Attention Network \(GAT\)as the teacher in several experiments\[[30](https://arxiv.org/html/2605.06814#bib.bib2),[48](https://arxiv.org/html/2605.06814#bib.bib60)\]\. Both models belong to the class of message\-passing neural networks, which learn node representations𝐡v\\mathbf\{h\}\_\{v\}used for predictionyv=f​\(𝐡v\)y\_\{v\}=f\(\\mathbf\{h\}\_\{v\}\)\. These representations are computed via iterative𝖠𝖦𝖦𝖱𝖤𝖦𝖠𝖳𝖤\\mathsf\{AGGREGATE\}and𝖢𝖮𝖬𝖡𝖨𝖭𝖤\\mathsf\{COMBINE\}operations:

𝐡vt\+1=𝖢𝖮𝖬𝖡𝖨𝖭𝖤t​\(𝐡vt,𝖠𝖦𝖦𝖱𝖤𝖦𝖠𝖳𝖤t​\(\{𝐡ut:Au​v=1\}\)\),\\mathbf\{h\}\_\{v\}^\{t\+1\}=\\mathsf\{COMBINE\}^\{t\}\\Big\(\\mathbf\{h\}\_\{v\}^\{t\},\\;\\mathsf\{AGGREGATE\}^\{t\}\\big\(\\\{\\mathbf\{h\}\_\{u\}^\{t\}:A\_\{uv\}=1\\\}\\big\)\\Big\),where𝐡vt\\mathbf\{h\}\_\{v\}^\{t\}denotes the embedding of nodevvat layertt\.

The primary difference between GCN and GAT lies in the aggregation mechanism\. GCN employs fixed, normalized adjacency weights to average neighbor features, whereas GAT learns attention coefficients that assign different importance to neighbors, enabling adaptive and data\-dependent message passing\.

For GCN, the layer\-wise update is given by:

𝐡vt\+1=σ​\(1\|N​\(v\)\|\+1​𝐖t​\(𝐡vt\|N​\(v\)\|\+1\+∑u∈N​\(v\)𝐡ut\|N​\(u\)\|\+1\)\),\\mathbf\{h\}\_\{v\}^\{t\+1\}=\\sigma\\left\(\\frac\{1\}\{\\sqrt\{\|N\(v\)\|\+1\}\}\\mathbf\{W\}^\{t\}\\left\(\\frac\{\\mathbf\{h\}\_\{v\}^\{t\}\}\{\\sqrt\{\|N\(v\)\|\+1\}\}\+\\sum\_\{u\\in N\(v\)\}\\frac\{\\mathbf\{h\}\_\{u\}^\{t\}\}\{\\sqrt\{\|N\(u\)\|\+1\}\}\\right\)\\right\),
For GAT, the update rule is:

𝐡vt\+1=σ​\(∑u∈N​\(v\)∪\{v\}αv​u​𝐖𝐡ut\),\\mathbf\{h\}\_\{v\}^\{t\+1\}=\\sigma\\left\(\\sum\_\{u\\in N\(v\)\\cup\\\{v\\\}\}\\alpha\_\{vu\}\\,\\mathbf\{W\}\\mathbf\{h\}\_\{u\}^\{t\}\\right\),whereαv​u\\alpha\_\{vu\}denotes the attention coefficient on edge\(v,u\)\(v,u\):

αv​u=exp⁡\(LeakyReLU​\(𝐚⊤​\[𝐡vt​𝐖,𝐡ut​𝐖\]\)\)∑k∈N​\(v\)∪\{v\}exp⁡\(LeakyReLU​\(𝐚⊤​\[𝐡vt​𝐖,𝐡kt​𝐖\]\)\),\\alpha\_\{vu\}=\\frac\{\\exp\\Big\(\\mathrm\{LeakyReLU\}\\big\(\\mathbf\{a\}^\{\\top\}\[\\mathbf\{h\}\_\{v\}^\{t\}\\mathbf\{W\},\\;\\mathbf\{h\}\_\{u\}^\{t\}\\mathbf\{W\}\]\\big\)\\Big\)\}\{\\sum\_\{k\\in N\(v\)\\cup\\\{v\\\}\}\\exp\\Big\(\\mathrm\{LeakyReLU\}\\big\(\\mathbf\{a\}^\{\\top\}\[\\mathbf\{h\}\_\{v\}^\{t\}\\mathbf\{W\},\\;\\mathbf\{h\}\_\{k\}^\{t\}\\mathbf\{W\}\]\\big\)\\Big\)\},with learnable parameters𝐖\\mathbf\{W\}and𝐚\\mathbf\{a\}\.

We also consider aGraph Transformer \(GT\)as a teacher model\. More specifically, we consider Graphormer\[[59](https://arxiv.org/html/2605.06814#bib.bib81)\]as a representative GT\. GTs are built by stacking multiple Transformer layers\[[47](https://arxiv.org/html/2605.06814#bib.bib86)\], where each layer comprises a self\-attention module followed by a feed\-forward network\.

Let𝐇∈ℝn×d\\mathbf\{H\}\\in\\mathbb\{R\}^\{n\\times d\}denote the input to the self\-attention module\. This input is linearly projected into query, key, and value representations via learnable matrices𝐖Q\\mathbf\{W\}\_\{Q\},𝐖K\\mathbf\{W\}\_\{K\}, and𝐖V\\mathbf\{W\}\_\{V\}:

𝐐=𝐇𝐖Q,𝐊=𝐇𝐖K,𝐕=𝐇𝐖V\.\\displaystyle\\mathbf\{Q\}=\\mathbf\{H\}\\mathbf\{W\}\_\{Q\},\\quad\\mathbf\{K\}=\\mathbf\{H\}\\mathbf\{W\}\_\{K\},\\quad\\mathbf\{V\}=\\mathbf\{H\}\\mathbf\{W\}\_\{V\}\.
The self\-attention operation is then computed as:

Attention​\(𝐐,𝐊,𝐕\)=softmax​\(𝐐𝐊⊤dK\)​𝐕,\\displaystyle\\mathrm\{Attention\}\(\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\)=\\mathrm\{softmax\}\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\_\{K\}\}\}\\right\)\\mathbf\{V\},wheredKd\_\{K\}denotes the dimensionality of the key vectors\.

## Evaluation Metrics

Utility:We use accuracy to evaluate utility\.

Fairness:We measure group fairness with respect to a sensitive attributess, wheres=1s=1denotes the sensitive group ands=0s=0the non\-sensitive group\. Lower values indicate better fairness\.

Demographic Parity \(DP\)measures the difference in positive prediction rates across groups:

DP\(y^,s\)=\|p\(y^=1∣s=1\)−p\(y^=1∣s=0\)\|\.DP\(\\hat\{y\},s\)=\\left\|p\(\\hat\{y\}=1\\mid s=1\)\-p\(\\hat\{y\}=1\\mid s=0\)\\right\|\.\(6\)
Equality of Opportunity \(EQOP\)measures the difference in true positive rates:

EQOP\(y^,s,y\)=\|p\(y^=1∣s=1,y=1\)−p\(y^=1∣s=0,y=1\)\|\.EQOP\(\\hat\{y\},s,y\)=\\left\|p\(\\hat\{y\}=1\\mid s=1,y=1\)\-p\(\\hat\{y\}=1\\mid s=0,y=1\)\\right\|\.\(7\)DP and EQOP both measure bias\. We can get corresponding fairness by subtracting them from 1\.

### Dataset Details

#### Dataset for Fairness Evaluation:

Simulation: We generate a synthetic graph with 1000 nodes using a stochastic block model with two communities of sizes 600 \(majority\) and 400 \(minority\), and set intra and inter\-block edge probabilities to achieve high assortativity \(≈0\.77\\approx 0\.77\)\. The sensitive attributessis generated conditionally on the label𝐲\\mathbf\{y\}:s∼Bernoulli​\(p\)s\\sim\\text\{Bernoulli\}\(p\)ify=1y=1ands∼Bernoulli​\(1−p\)s\\sim\\text\{Bernoulli\}\(1\-p\)otherwise, we setp=0\.7p=0\.7to induce bias, creating high correlation betweenssand𝐲\\mathbf\{y\}\. This setup introduces structural bias, as nodes are more likely to connect within the same community, leading to label–group correlations in the graph topology\. Each node has 20 features, of which 8 are noisy, sampled from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\), while the remaining features are generated asγ⋅y\+𝒩​\(0,1\)\\gamma\\cdot y\+\\mathcal\{N\}\(0,1\)\. This introduces feature\-level bias, as informative attributes are correlated with the label and, indirectly, with the sensitive attribute, reinforcing disparities present in both structure and features\.

German\[[23](https://arxiv.org/html/2605.06814#bib.bib82)\]: Vertices represent 1000 individuals with 20 financial and demographic features\. Edges are computed from vertex similarity\. The prediction task is to classify credit \(good vs\. bad credit\), with gender treated as the sensitive attribute\.

NBA\[[6](https://arxiv.org/html/2605.06814#bib.bib78)\]: This dataset consists of 403 NBA players connected via their Twitter interactions\. Node features capture player statistics, and the task is to predict whether a player’s salary is above or below the median\. Nationality is treated as a sensitive attribute\.

#### Dataset for Improved Performance Evaluation

The statistics of the datasets used to analyze the performance of GAT and GT are provided in Table[2](https://arxiv.org/html/2605.06814#Ax2.T2)\.

### Baselines

Vanilla: The vanilla model is the student model without any distillation\.

FairGNN\[[6](https://arxiv.org/html/2605.06814#bib.bib78)\]: Jointly trains a GNN classifier with a neural network adversary, where the adversary encourages the learned representations to remove sensitive information and thus mitigate bias\.

FairVGNN\[[51](https://arxiv.org/html/2605.06814#bib.bib79)\]: FairVGNN learns fair feature views by identifying and masking features correlated with sensitive attributes, explicitly accounting for shifts in these correlations induced by feature propagation\. Conditioned on the resulting views, it adaptively regularizes the encoder parameters to suppress the influence of sensitive\-related information\.

NIFTY\[[1](https://arxiv.org/html/2605.06814#bib.bib80)\]: NIFTY ensures fairness and stability by combining a joint fairness–stability objective with Lipschitz\-based layer\-wise weight normalization\. It provides theoretical guarantees for counterfactual fairness and stable representations, and achieves strong empirical performance on high\-stakes datasets\.

GT\[[59](https://arxiv.org/html/2605.06814#bib.bib81)\]: Transformer\-based model for graph representation learning that achieves strong performance across a wide range of tasks\. Its core design centers on explicitly encoding graph structure into the Transformer architecture through centrality encoding, edge encoding, and spatial encoding, enabling effective modeling of node relationships beyond standard message passing\.

Table 2:Dataset statistics\[[42](https://arxiv.org/html/2605.06814#bib.bib84),[43](https://arxiv.org/html/2605.06814#bib.bib85),[14](https://arxiv.org/html/2605.06814#bib.bib83)\]
### Proof of Theorems

Our theorems are based on feature augmentations\. A similar analysis can be provided for structural augmentations where the learned structure approximates the representations from the teacher model\.

### Theorem 1

###### Proof\.

Let us assume a linear teacher and student prediction head\. Then, let the teacher and student logits be as follows:

𝐲^T=𝐔T​𝐇T,𝐲^s=𝐔s​𝐇s\.\\hat\{\\mathbf\{y\}\}\_\{T\}=\\mathbf\{U\}\_\{T\}\\mathbf\{H\}\_\{T\},\\qquad\\hat\{\\mathbf\{y\}\}\_\{s\}=\\mathbf\{U\}\_\{s\}\\mathbf\{H\}\_\{s\}\.As‖𝐲^s−𝐲^T‖22≈0\\\|\\hat\{\\mathbf\{y\}\}\_\{s\}\-\\hat\{\\mathbf\{y\}\}\_\{T\}\\\|^\{2\}\_\{2\}\\approx 0,𝐔s​𝐇s=𝐔T​𝐇T\\mathbf\{U\}\_\{s\}\\mathbf\{H\}\_\{s\}=\\mathbf\{U\}\_\{T\}\\mathbf\{H\}\_\{T\}\. We obtain𝐇s\\mathbf\{H\}\_\{s\}by solving the following least\-squares problem:

min𝐇s⁡‖𝐔s​𝐇s−𝐔T​𝐇T‖F2\.\\min\_\{\\mathbf\{H\}\_\{s\}\}\\\|\\mathbf\{U\}\_\{s\}\\mathbf\{H\}\_\{s\}\-\\mathbf\{U\}\_\{T\}\\mathbf\{H\}\_\{T\}\\\|\_\{F\}^\{2\}\.Gradient descent with zero initialization converges to the minimum\-norm solution of the linear least\-squares problem\[[17](https://arxiv.org/html/2605.06814#bib.bib87)\]\. Therefore, we get:

𝐇s=𝐔s†​𝐔T​𝐇T,\\mathbf\{H\}\_\{s\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\mathbf\{U\}\_\{T\}\\mathbf\{H\}\_\{T\},where𝐔s†\\mathbf\{U\}\_\{s\}^\{\\dagger\}denotes the Moore–Penrose pseudo\-inverse\.

Hence,𝐇s\\mathbf\{H\}\_\{s\}is a linear \(and therefore measurable\) function of𝐇T\\mathbf\{H\}\_\{T\}\. Since𝐇T⟂𝐬\\mathbf\{H\}\_\{T\}\\perp\\mathbf\{s\}and measurable functions preserve independence, it follows that𝐇s⟂𝐬\\mathbf\{H\}\_\{s\}\\perp\\mathbf\{s\}\. Finally, let𝐙=fϕ​\(𝐇s\)\\mathbf\{Z\}=f\_\{\\phi\}\(\\mathbf\{H\}\_\{s\}\), wherefϕf\_\{\\phi\}is Borel measurable\. Then𝐙\\mathbf\{Z\}is a measurable function of𝐇s\\mathbf\{H\}\_\{s\}, and thus𝐙⟂𝐬\\mathbf\{Z\}\\perp\\mathbf\{s\}\. ∎

The theorem shows that M2D can always distill a fair model via feature augmentation alone based only on the teacher model’s logits\. However, we have shown empirically that this can be achieved based on a small number of features using the diversity regularization loss described in Section[2\.4](https://arxiv.org/html/2605.06814#S2.SS4), which is a stronger result\.

#### Theorem 2

###### Proof\.

Let us consider a single\-layer GAT, and the teacher embedding for nodeiibe as follows:

𝐡T,i=σ​\(∑j∈𝒩​\(i\)αi​j​𝐖𝐱j\),\\mathbf\{h\}\_\{T,i\}=\\sigma\\left\(\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{W\}\\mathbf\{x\}\_\{j\}\\right\),\(8\)whereσ​\(⋅\)\\sigma\(\\cdot\)is a pointwise nonlinearity \(e\.g\., ReLU or LeakyReLU\)\.

Using a first\-order Taylor expansion ofσ​\(⋅\)\\sigma\(\\cdot\), there exists a linear map𝐖~\\tilde\{\\mathbf\{W\}\}and a bias term𝐛i\\mathbf\{b\}\_\{i\}such that

𝐡T,i=∑j∈𝒩​\(i\)αi​j​𝐖~​𝐱j\+𝐛i\.\\mathbf\{h\}\_\{T,i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\tilde\{\\mathbf\{W\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{b\}\_\{i\}\.
Let the teacher and student logits be:

𝐲^T,i=𝐔T​𝐡T,i,𝐲^s,i=𝐔s​𝐡s,i\.\\hat\{\\mathbf\{y\}\}\_\{T,i\}=\\mathbf\{U\}\_\{T\}\\mathbf\{h\}\_\{T,i\},\\qquad\\hat\{\\mathbf\{y\}\}\_\{s,i\}=\\mathbf\{U\}\_\{s\}\\mathbf\{h\}\_\{s,i\}\.
Under the assumption of small distillation error, we have

‖𝐔s​𝐡s,i−𝐔T​𝐡T,i‖22=‖ϵi‖22≤ϵ\.\\\|\\mathbf\{U\}\_\{s\}\\mathbf\{h\}\_\{s,i\}\-\\mathbf\{U\}\_\{T\}\\mathbf\{h\}\_\{T,i\}\\\|\_\{2\}^\{2\}=\\\|\\boldsymbol\{\\epsilon\}\_\{i\}\\\|\_\{2\}^\{2\}\\leq\\epsilon\.where ,ϵi=𝐔s​𝐡s,i−𝐔T​𝐡T,i\\epsilon\_\{i\}=\\mathbf\{U\}\_\{s\}\\mathbf\{h\}\_\{s,i\}\-\\mathbf\{U\}\_\{T\}\\mathbf\{h\}\_\{T,i\}Therefore:

𝐔s​𝐡s,i=𝐔T​𝐡T,i\+ϵi\.\\mathbf\{U\}\_\{s\}\\mathbf\{h\}\_\{s,i\}=\\mathbf\{U\}\_\{T\}\\mathbf\{h\}\_\{T,i\}\+\\boldsymbol\{\\epsilon\}\_\{i\}\.
Substituting𝐡T,i\\mathbf\{h\}\_\{T,i\}:

𝐔s​𝐡s,i\\displaystyle\\mathbf\{U\}\_\{s\}\\mathbf\{h\}\_\{s,i\}=𝐔T​\(∑jαi​j​𝐖~​𝐱j\+𝐛i\)\+ϵi\\displaystyle=\\mathbf\{U\}\_\{T\}\\left\(\\sum\_\{j\}\\alpha\_\{ij\}\\tilde\{\\mathbf\{W\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{b\}\_\{i\}\\right\)\+\\boldsymbol\{\\epsilon\}\_\{i\}=∑jαi​j​𝐔T​𝐖~​𝐱j\+𝐔T​𝐛i\+ϵi\.\\displaystyle=\\sum\_\{j\}\\alpha\_\{ij\}\\mathbf\{U\}\_\{T\}\\tilde\{\\mathbf\{W\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{U\}\_\{T\}\\mathbf\{b\}\_\{i\}\+\\boldsymbol\{\\epsilon\}\_\{i\}\.
This is a linear system in𝐡s,i\\mathbf\{h\}\_\{s,i\}\. It follows that:

𝐡s,i=𝐔s†​\(∑jαi​j​𝐔T​𝐖~​𝐱j\+𝐔T​𝐛i\+ϵi\)\+𝐧i,\\mathbf\{h\}\_\{s,i\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\left\(\\sum\_\{j\}\\alpha\_\{ij\}\\mathbf\{U\}\_\{T\}\\tilde\{\\mathbf\{W\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{U\}\_\{T\}\\mathbf\{b\}\_\{i\}\+\\boldsymbol\{\\epsilon\}\_\{i\}\\right\)\+\\mathbf\{n\}\_\{i\},\(9\)where𝐔s†\\mathbf\{U\}\_\{s\}^\{\\dagger\}is the Moore–Penrose pseudoinverse and𝐧i∈Null​\(𝐔s\)\\mathbf\{n\}\_\{i\}\\in\\mathrm\{Null\}\(\\mathbf\{U\}\_\{s\}\)\.

Rearranging the terms:

𝐡s,i\\displaystyle\\mathbf\{h\}\_\{s,i\}=∑jαi​j​𝐔s†​𝐔T​𝐖~​𝐱j\+𝐔s†​\(𝐔T​𝐛i\+ϵi\)\+𝐧i\.\\displaystyle=\\sum\_\{j\}\\alpha\_\{ij\}\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\mathbf\{U\}\_\{T\}\\tilde\{\\mathbf\{W\}\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{U\}\_\{s\}^\{\\dagger\}\(\\mathbf\{U\}\_\{T\}\\mathbf\{b\}\_\{i\}\+\\boldsymbol\{\\epsilon\}\_\{i\}\)\+\\mathbf\{n\}\_\{i\}\.
Let’s define the following:

𝐌=𝐔s†​𝐔T​𝐖~,𝐫i=𝐔s†​\(𝐔T​𝐛i\+ϵi\)\.\\mathbf\{M\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\mathbf\{U\}\_\{T\}\\tilde\{\\mathbf\{W\}\},\\qquad\\mathbf\{r\}\_\{i\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\(\\mathbf\{U\}\_\{T\}\\mathbf\{b\}\_\{i\}\+\\boldsymbol\{\\epsilon\}\_\{i\}\)\.
Then, we get that:

𝐡s,i=∑j∈𝒩​\(i\)αi​j​𝐌𝐱j\+𝐫i\+𝐧i,\\mathbf\{h\}\_\{s,i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{r\}\_\{i\}\+\\mathbf\{n\}\_\{i\},where𝐧i∈Null​\(𝐔s\)\\mathbf\{n\}\_\{i\}\\in\\mathrm\{Null\}\(\\mathbf\{U\}\_\{s\}\)andrir\_\{i\}captures the residual term arising from the linearization bias of GAT and distillation error\.

Applying the feature learnerfϕf\_\{\\phi\}, we get:

𝐳i=fϕ​\(𝐡s,i\)=fϕ​\(∑j∈𝒩​\(i\)αi​j​𝐌𝐱j\+𝐫i\+𝐧i\),\\mathbf\{z\}\_\{i\}=f\_\{\\phi\}\\left\(\\mathbf\{h\}\_\{s,i\}\\right\)=f\_\{\\phi\}\(\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{r\}\_\{i\}\+\\mathbf\{n\}\_\{i\}\),∎

Similar to Theorem[1](https://arxiv.org/html/2605.06814#Thmtheorem1), the theorem shows that M2D can always distill GAT representations via feature augmentation alone based only on the teacher model’s logits\. However, we have shown empirically that this can be achieved based on a small number of features using the diversity regularization loss described in Section[2\.4](https://arxiv.org/html/2605.06814#S2.SS4)\. A similar proof can be derived for GT\.

#### Corollary 1

###### Proof\.

The attention coefficients are given by a softmax over scoresei​je\_\{ij\}, hence:

αi​j\>αi​k⟹ei​j\>ei​k\.\\alpha\_\{ij\}\>\\alpha\_\{ik\}\\;\\Longrightarrow\\;e\_\{ij\}\>e\_\{ik\}\.By the homophily condition,ψ\\psiis monotone, then:

sim​\(𝐡T,i,𝐡T,j\)\>sim​\(𝐡T,i,𝐡T,k\)\.\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,j\}\)\>\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,k\}\)\.Applying𝐫i≈0\\mathbf\{r\}\_\{i\}\\approx 0, let us define the attention\-weighted aggregated representation as:

𝐮i=∑j∈𝒩​\(i\)αi​j​𝐌𝐱j\+𝐧i,\\mathbf\{u\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}\+\\mathbf\{n\}\_\{i\},If we use gradient descent with zero initialization to optimize the distillation, then𝐧i=𝟎\\mathbf\{n\}\_\{i\}=\\mathbf\{0\}as𝐡s,i\\mathbf\{h\}\_\{s,i\}in Equation[9](https://arxiv.org/html/2605.06814#Ax2.E9)converges to the minimum\-norm solution of the linear least square problem\[[17](https://arxiv.org/html/2605.06814#bib.bib87)\]\. Therefore𝐮i=∑j∈𝒩​\(i\)αi​j​𝐌𝐱j\\mathbf\{u\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}, and the corresponding learned feature is𝐳i=fϕ​\(𝐮i\)\\mathbf\{z\}\_\{i\}=f\_\{\\phi\}\(\\mathbf\{u\}\_\{i\}\)\.

Using the linear form of the teacher representation𝐡T,i=∑j∈𝒩​\(i\)αi​j​𝐖𝐱j\\mathbf\{h\}\_\{T,i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{W\}\\mathbf\{x\}\_\{j\}and𝐌=𝐔s†​𝐔T​𝐖\\mathbf\{M\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\mathbf\{U\}\_\{T\}\\mathbf\{W\}, we can write:

𝐮i=∑j∈𝒩​\(i\)αi​j​𝐌𝐱j=𝐔s†​𝐔T​𝐡T,i,\\mathbf\{u\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{N\}\(i\)\}\\alpha\_\{ij\}\\mathbf\{M\}\\mathbf\{x\}\_\{j\}=\\mathbf\{U\}\_\{s\}^\{\\dagger\}\\mathbf\{U\}\_\{T\}\\mathbf\{h\}\_\{T,i\},
Thus,𝐮i\\mathbf\{u\}\_\{i\}is a linear transformation of the teacher representation𝐡T,i\\mathbf\{h\}\_\{T,i\}\. Assuming that the similarity measure is order\-preserving under this linear transformation, we obtain:

sim​\(𝐡T,i,𝐡T,j\)\>sim​\(𝐡T,i,𝐡T,k\)⟹sim​\(𝐮i,𝐮j\)\>sim​\(𝐮i,𝐮k\)\.\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,j\}\)\>\\mathrm\{sim\}\(\\mathbf\{h\}\_\{T,i\},\\mathbf\{h\}\_\{T,k\}\)\\;\\Longrightarrow\\;\\mathrm\{sim\}\(\\mathbf\{u\}\_\{i\},\\mathbf\{u\}\_\{j\}\)\>\\mathrm\{sim\}\(\\mathbf\{u\}\_\{i\},\\mathbf\{u\}\_\{k\}\)\.Finally, since𝐳i=fϕ​\(𝐮i\)\\mathbf\{z\}\_\{i\}=f\_\{\\phi\}\(\\mathbf\{u\}\_\{i\}\)andfϕf\_\{\\phi\}is order\-preserving with respect tosim​\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\), it follows that

sim​\(𝐳i,𝐳j\)\>sim​\(𝐳i,𝐳k\)\.\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)\>\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{k\}\)\.∎

### Training Algorithm

Algorithm[1](https://arxiv.org/html/2605.06814#alg1)shows the training of M2D\. M2D performs iterative distillation by jointly refining the student model and the underlying graph structure\. At each iteration, the current student representation is used to generate an augmented graph\. The student is then updated using this augmented input to match the teacher outputs\. Subsequently, the graph and feature learners are updated based on the refined student representation\. This alternating process enables mutual improvement between representation learning and structure adaptation\. The final augmented graph and features capture the teacher’s behavior more effectively\.

Algorithm 1M2D Distillation0:Graph

𝐀\\mathbf\{A\}, features

𝐗\\mathbf\{X\}, labels

𝐲\\mathbf\{y\}, teacher outputs

𝐲^T\\hat\{\\mathbf\{y\}\}\_\{T\}
1:Initialize

Θs0\\Theta\_\{s\}^\{0\},

Θg0\\Theta\_\{g\}^\{0\}and

𝐇s0←fs​\(𝐗,𝐀\)\\mathbf\{H\}\_\{s\}^\{0\}\\leftarrow f\_\{s\}\(\\mathbf\{X\},\\mathbf\{A\}\)
2:for

t=1t=1to

tmaxt\_\{\\max\}do

3:\(1\) Construct augmented graph and features

4:

𝐙t←fϕ​\(𝐇st−1\)\\mathbf\{Z\}^\{t\}\\leftarrow f\_\{\\phi\}\(\\mathbf\{H\}\_\{s\}^\{t\-1\}\),

𝐗~t←\[𝐗∥𝐙t\]\\tilde\{\\mathbf\{X\}\}^\{t\}\\leftarrow\[\\mathbf\{X\}\\,\\\|\\,\\mathbf\{Z\}^\{t\}\]
5:

𝐖t←fa​\(𝐇st−1\)\\mathbf\{W\}^\{t\}\\leftarrow f\_\{a\}\(\\mathbf\{H\}\_\{s\}^\{t\-1\}\)
6:

𝐀~t←\(1−α\)​𝐀\+α​\(\(1−β\)​𝐖t\+β​s​\(𝐗\)\)\\tilde\{\\mathbf\{A\}\}^\{t\}\\leftarrow\(1\-\\alpha\)\\mathbf\{A\}\+\\alpha\\big\(\(1\-\\beta\)\\mathbf\{W\}^\{t\}\+\\beta\\,s\(\\mathbf\{X\}\)\\big\)
7:\(2\) Update student \(inner step\)

8:

𝐇st←fs​\(𝐗~t,𝐀~t\)\\mathbf\{H\}\_\{s\}^\{t\}\\leftarrow f\_\{s\}\(\\tilde\{\\mathbf\{X\}\}^\{t\},\\tilde\{\\mathbf\{A\}\}^\{t\}\)
9:

𝐲^st←fc​\(𝐇st\)\\hat\{\\mathbf\{y\}\}\_\{s\}^\{t\}\\leftarrow f\_\{c\}\(\\mathbf\{H\}\_\{s\}^\{t\}\)
10:Update

Θst\\Theta\_\{s\}^\{t\}using

ℒc​l​s\+ℒd​i​s\\mathcal\{L\}\_\{cls\}\+\\mathcal\{L\}\_\{dis\}
11:\(3\) Update graph and feature learner \(outer step\)

12:

𝐙t←fϕ​\(𝐇st\)\\mathbf\{Z\}^\{t\}\\leftarrow f\_\{\\phi\}\(\\mathbf\{H\}\_\{s\}^\{t\}\),

𝐖t←fa​\(𝐇st\)\\mathbf\{W\}^\{t\}\\leftarrow f\_\{a\}\(\\mathbf\{H\}\_\{s\}^\{t\}\)
13:Update

Θgt\\Theta\_\{g\}^\{t\}using

ℒc​l​s\+ℒd​i​s\+ℒd​i​v\+ℒg​r​a​p​h\\mathcal\{L\}\_\{cls\}\+\\mathcal\{L\}\_\{dis\}\+\\mathcal\{L\}\_\{div\}\+\\mathcal\{L\}\_\{graph\}
14:endfor

15:return

𝐀~t\\tilde\{\\mathbf\{A\}\}^\{t\},

𝐗~t\\tilde\{\\mathbf\{X\}\}^\{t\}

## Software and Hardware

- •Operating System: Linux \(Red Hat Enterprise Linux 8\.9 \(Ootpa\)\)
- •GPU: NVIDIA A40
- •Software: Python 3\.8\.10, torch 2\.2\.1

Table 3:Distillation accuracy of different M2D variants for fair teacherTable 4:Distillation accuracy of different M2D variants for the GAT and GT teacher### Experimental Setup

All experiments use two\-dimensional learned features for all datasets, exceptNBA, where a single feature is learned\. All GNN models consist of two layers\. We use a linear prediction head for both student and teacher in all experiments\. For Graph Transformer experiments, we adopt Graphormer\[[59](https://arxiv.org/html/2605.06814#bib.bib81)\]\. All reported results are averaged over55independent runs with different random seeds\. Hyperparameters are selected via grid search based on validation performance\. To mitigate overfitting, we employ early stopping after150150iterations\. Each dataset is divided into 30% for training, 20% for validation, and 50% for testing\. We use the Adam optimizer to train all model parameters\.

### Additional Results and Visualization

Table[3](https://arxiv.org/html/2605.06814#Ax3.T3)reports the distillation accuracy of different M2D variants for fair GNN teachers\. Overall, M2D consistently improves the student’s ability to approximate the teacher compared to the baseline GCN without augmentation\. In most settings, distillation with both structure and feature achieves the best or near\-best performance, indicating that jointly adapting node features and graph structure improves distillation\. Feature based augmentation also performs competitively, often outperforming adjacency\-only augmentation\. These results show that M2D effectively transfers the behavior of fairness aware teacher models to simpler student architectures\.

Table[4](https://arxiv.org/html/2605.06814#Ax3.T4)reports the distillation accuracy across different M2D variants and teachers\. Overall, GCN\-based students achieve consistently high accuracy when distilling from both GAT and GT teachers on homophilic datasets \(Cora, Citeseer, Photo\), indicating low distillation error and effective transfer of teacher behavior\. In contrast, MLP\-based students exhibit significantly lower accuracy on these datasets, suggesting that the absence of structural information limits their ability to match the teacher\. On heterophilic datasets \(Cornell, Texas\), the trend differs: MLP students achieve higher accuracy, particularly when distilling from GT, while GCN performance degrades in certain settings\. These results highlight that distillation accuracy and, consequently, the ability to capture teacher behavior depend strongly on the compatibility between the student architecture and the underlying graph structure\.

Table[5](https://arxiv.org/html/2605.06814#Ax3.T5)compares node classification accuracy of teacher models \(GAT, GT\) and student MLP under different distillation settings\. On homophilic datasets such asCora,Citeseer, andPhoto, M2D consistently improves student performance over both vanilla training and standard distillation, demonstrating effective knowledge transfer from the teacher\. However, on heterophilic datasets such asCornellandTexas, the vanilla MLP already outperforms the teacher models\. In these cases, M2D leads to performance degradation, as the student is encouraged to mimic suboptimal teacher representations\. This highlights that the effectiveness of distillation depends on the quality of the teacher and that blindly transferring knowledge can be detrimental when the teacher is weaker than the student\.

Figure[9](https://arxiv.org/html/2605.06814#Ax3.F9)visualizes the two dimensions of the learned features across five benchmark datasets\. Each point corresponds to a node and is colored by its class label\. Across homophilic datasets such asCora,Citeseer, andPhoto, the learned features exhibit clear class\-wise clustering, indicating that the student effectively captures discriminative structure from the teacher\. In contrast, the MLP produces less structured embeddings, with weaker separation between classes, highlighting the importance of graph\-based inductive bias\. On heterophilic datasets such asCornellandTexas, the clustering structure is less pronounced, reflecting the inherent difficulty of these datasets and the weaker supervision signal from the teacher\. Overall, these visualizations qualitatively support our quantitative results, showing that M2D leads to more structured and class\-discriminative representations when the teacher provides meaningful knowledge\. We observe similar trends for Graph Transformer \(GT\) teachers \(Figure[10](https://arxiv.org/html/2605.06814#Ax3.F10)\)\. The learned representations again exhibit clear class\-wise clustering on homophilic datasets\. The MLP baseline remains less structured, and clustering degrades on heterophilic datasets such asCornellandTexas\.

Table 5:Node classification accuracy comparing teacher models \(GAT, GT\) and student MLP under different settings\. ‘Van\.’ denotes the student trained without distillation, ‘w/o’ standard distillation, and ‘Feat\.’ M2D feature distillation\. M2D improves student performance onCora,Citeseer, andPhoto\. OnCornellandTexas, where the teacher underperforms the vanilla MLP, M2D degrades performance by transferring suboptimal knowledge\.![Refer to caption](https://arxiv.org/html/2605.06814v1/x9.png)Figure 9:Learned features visualization across five benchmark datasets \(Cora, Citeseer, Photo, Cornell, and Texas\) obtained from the GAT teacher\. The first and second rows present the features learned by the student GCN using onlyfϕf\_\{\\phi\}and using bothfϕf\_\{\\phi\}andfaf\_\{a\}, respectively, while the third row shows the features produced by the student MLP\. Each point corresponds to a node, colored by its class label, illustrating the separability and clustering behavior of the learned features\.![Refer to caption](https://arxiv.org/html/2605.06814v1/x10.png)Figure 10:Learned features visualization across five benchmark datasets \(Cora, Citeseer, Photo, Cornell, and Texas\) obtained from the GT teacher\. The first and second rows present the features learned by the student GCN using onlyfϕf\_\{\\phi\}and using bothfϕf\_\{\\phi\}andfaf\_\{a\}, respectively, while the third row shows the features produced by the student MLP\. Each point corresponds to a node, colored by its class label, illustrating the separability and clustering behavior of the learned features\.Figure[11](https://arxiv.org/html/2605.06814#Ax3.F11)illustrates the effect of feature\-based distillation onCora\. The learned features form well\-defined clusters corresponding to class labels\. We highlight nodes that are misclassified by standard GCN distillation but correctly classified by GCN\(Feat\.\)\. These nodes are primarily located near the boundaries between clusters\. While standard GCN fails to correctly classify these boundary nodes, GCN\(Feat\.\) produces representations that group nodes according to their class labels, enabling predictions that are consistent with the teacher\. This shows that feature\-based distillation helps recover the correct class structure, particularly for nodes near class boundaries\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x11.png)Figure 11:Learned feature visualization onCorawith GAT as teacher and GCN as student\. Black circles denote nodes misclassified by standard GCN distillation but correctly classified by GCN\(Feat\.\)\. The learned features form clear class\-wise clusters, and these nodes lie near cluster boundaries, where GCN\(Feat\.\) successfully matches the teacher while standard GCN fails\.![Refer to caption](https://arxiv.org/html/2605.06814v1/x12.png)Figure 12:Visualization of edge importance on Cornell \(heterophilic\) using row\-normalized M2D edge weights \(left\) and layer\-1 GAT attention coefficients \(right\)\. Edge color and width indicate magnitude, with darker and thicker edges denoting higher importance\. Learned weights are row\-normalized \(sum to one\), enabling direct probabilistic comparison with attention\. Both panels share an identical layout\. The left panel highlights how M2D reweights edges to approximate GAT behavior\. However, the Pearson correlation coefficient between the edge weights is0\.110\.11\(low\)\.Figure[12](https://arxiv.org/html/2605.06814#Ax3.F12)compares edge importance from M2D and GAT on the heterophilicCornellgraph using a shared layout\. While GAT directly learns attention coefficients, M2D learns edge weights that we row\-normalize for comparison, enabling a probabilistic interpretation\. The visualization reveals clear differences in how edges are weighted across the two methods\. Due to heterophily, the normalized M2D weights do not match GAT attention exactly, but instead reflect the adjustments required for the student to approximate the teacher’s behavior\. This highlights the role of edge reweighting in aligning the student with attention\-based representations\. The difference in scale between the two plots is due to the hyperparameterγ\\gamma\. For theCornelldataset, a higher value of the hyperparameter \(0\.45\) generates the best result\.

![Refer to caption](https://arxiv.org/html/2605.06814v1/x13.png)Figure 13:Analysis of learned features with respect to GT attention\. We bin feature similaritysim​\(𝐳i,𝐳j\)\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)and compute average attention per bin\. GCN students exhibit a clear increasing relationship in homophilic datasets \(Cora, Citeseer, Photo\), indicating alignment with teacher attention\. This trend weakens in heterophilic datasets due to lower distillation accuracy\. For MLP students, the trend appears only when distillation accuracy is high\.In Figure[13](https://arxiv.org/html/2605.06814#Ax3.F13), we analyze how learned features align with the attention mechanism of the GT teacher by examining the relationship between feature similarity and attention weights \(Corollary[1](https://arxiv.org/html/2605.06814#Thmcorollary1)\)\. Specifically, we partition node pairs into bins based on similaritysim​\(𝐳i,𝐳j\)\\mathrm\{sim\}\(\\mathbf\{z\}\_\{i\},\\mathbf\{z\}\_\{j\}\)and compute the average attention within each bin \(Similar to Figure[7](https://arxiv.org/html/2605.06814#S4.F7)\)\. In homophilic datasets \(Cora, Citeseer, Photo\), GCN students exhibit a clear positive relationship, indicating that higher attention weights correspond to more similar learned features, consistent with Corollary[1](https://arxiv.org/html/2605.06814#Thmcorollary1)\. In contrast, this relationship weakens in heterophilic datasets \(Cornell, Texas\), due to lower distillation accuracy \(See Table[4](https://arxiv.org/html/2605.06814#Ax3.T4)\)\. For MLP students, the trend is absent in homophilic datasets due to poor distillation performance but emerges in heterophilic settings where the student better matches the teacher\. These results indicate that higher feature similarity is associated with higher attention weights when distillation accuracy is sufficiently high\. Though in some cases, the feature similarity is not consistent with attention, performance still improves because the learned features are informative and cluster the classes based on labels \(see Figure[10](https://arxiv.org/html/2605.06814#Ax3.F10)\)\.

### Broader Impacts

This work improves the transparency of complex graph neural networks by using M2D\. The case study on fair GNNs is particularly relevant, as it demonstrates how M2D can make these models more transparent to humans, increasing their applicability to high\-stakes scenarios\. This can aid understanding and analysis of how complex models make decisions\. However, the method does not guarantee fairness and may not address biases in the data or teacher models\. Similar to model distillation approaches, M2D can be used to disclose information about the teacher model\. More specifically, one can use M2D to reverse\-engineer the teacher model from its logits\.

### Reproducibility

Our code is available in the anonymous GitHub repository https://anonymous\.4open\.science/r/m2d\-1F9A