MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

# MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion
Source: [https://arxiv.org/html/2607.15592](https://arxiv.org/html/2607.15592)
,Meiyu LiangSchool of Computer Science, Beijing University of Posts and TelecommunicationsBeijingChina,Wei HuangZhejiang UniversityHangzhouChinaBeijing University of Posts and TelecommunicationsBeijingChina,Yawen LiBeijing University of Posts and TelecommunicationsBeijingChina,Zhe XueSchool of Computer Science, Beijing University of Posts and TelecommunicationsBeijingChina,Wu LiuUniversity of Science and Technology of ChinaHefeiChina,Guanhua YeSchool of Computer Science \(National Pilot Software Engineering School\), Beijing University of Posts and TelecommunicationsBeijingChina,Lei ShiCommunication University of ChinaBeijingChinaandKangkang LuBeijing University of Posts and TelecommunicationsBeijingChina

\(2026\)

###### Abstract\.

Multimodal Knowledge Graph Completion \(MKGC\) requires inferring missing entities from structural, textual, and visual cues\. Existing diffusion\-based MKGC methods usually denoise directly on raw multimodal features\. Such a design forces the denoiser to simultaneously perform relation\-dependent cue selection, cross\-modal semantic alignment, and structure\-aware entity generation, which introduces noisy and semantically inconsistent conditions for diffusion and consequently leads to suboptimal completion performance\. To address this limitation, we proposeM2GDT:MLLM\-GuidedDiffusionTransformer with Relation\-AdaptiveMixture\-of\-Experts \(M2GDT\), a novel MKGC framework built on analign\-then\-diffuseparadigm\. M2GDT first employs aRelation\-AdaptiveSemanticRoutingMixture\-of\-Experts \(RASR\-MoE\) module to select relation\-relevant multimodal semantic transformation paths and suppress irrelevant modality interference\. M2GDT then uses a frozen Multimodal Large Language Model \(MLLM\) as a semantic anchor to align the routed multimodal representations into a unified latent space and reduce cross\-modal semantic heterogeneity\. Finally, aKnowledgeGraphDiffusionTransformer \(KGDT\) performs graph\-conditioned denoising generation in the aligned space to produce the missing entity representation\. Experiments on three benchmark datasets show that M2GDT consistently outperforms strong baselines\.

Multimodal Knowledge Graph, Multimodal Language Model, Diffusion Model, Mixture\-of\-Experts

††copyright:acmlicensed††journalyear:2026††doi:XXXXXXX\.XXXXXXX††conference:Proceedings of the 34th ACM International Conference on Multimedia; 10–14 November 2026; Rio de Janeiro, Brazil††isbn:978\-1\-4503\-XXXX\-X/2018/06††ccs:Computing methodologies Reasoning about belief and knowledge## 1\.Introduction

![Refer to caption](https://arxiv.org/html/2607.15592v1/figures/mkgc_challenge.png)Figure 1\.Core challenges of MKGC\. Structural, textual, and visual features exhibit cross\-modal semantic heterogeneity and relation\-dependent multimodal cue selection\. Solid lines indicate intra\-modal relations, while dashed lines indicate inter\-modal relations\. As a result, raw multimodal representations do not form a reliable unified reasoning space for relation\-conditioned entity completion\.Multimodal Knowledge Graph Completion \(MKGC\) aims to infer missing entities or links in a knowledge graph by jointly leveraging structural relations, textual descriptions, and visual cues\. Compared with conventional knowledge graph completion, MKGC is more practical for real\-world scenarios, where entities are often associated with rich multimodal attributes beyond symbolic triples\. In this setting, the model must answer relation\-conditioned queries such as\(h,r,?\)\(h,r,?\)or\(?,r,t\)\(?,r,t\)by reasoning over graph structure together with heterogeneous multimodal observations\. This makes MKGC substantially more challenging than unimodal completion, since the model must not only identify the correct entity but also ensure that the completed triple remains semantically meaningful and structurally consistent\.

Specifically, MKGC is mainly challenged by two coupled difficulties\. The first iscross\-modal semantic heterogeneity: structural, textual, and visual features are produced by different encoders and optimization objectives, causing them to reside in incompatible semantic spaces\. Even when describing the same entity, different modalities often emphasize different semantic aspects, resulting in substantial cross\-modal semantic gaps\. As illustrated in Figure[1](https://arxiv.org/html/2607.15592#S1.F1), this makes raw multimodal representations neither directly comparable nor reliably composable as a unified reasoning substrate\. The second difficulty isrelation\-dependent multimodal cues selection: different relations often rely on different semantic cues\. Some cases are mainly resolved by structural context, while others depend more heavily on textual semantics or visual cues\. Thus, effective MKGC requires not only cross\-modal alignment, but also relation\-aware use of multimodal knowledge before completion\.

Existing discriminative methods address these two challenges from different perspectives\. Early approaches such as IKRL\(Xieet al\.,[2017](https://arxiv.org/html/2607.15592#bib.bib123)\)and TransAE\(Wanget al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib155)\)primarily alleviate the first challenge by projecting multimodal attributes into a structural embedding space for triple scoring\. More recent methods further improve multimodal representation learning for semantically heterogeneous inputs\. For example, OTKGE\(Caoet al\.,[2022](https://arxiv.org/html/2607.15592#bib.bib152)\)addresses cross\-modal inconsistency more explicitly by introducing optimal transport to better align multimodal representations, while MyGo\(Zhanget al\.,[2025](https://arxiv.org/html/2607.15592#bib.bib165)\)improves multimodal entity modeling through discrete codebook, yielding richer multimodal representations for ranking\. In contrast, MoMoK\(Zhanget al\.,[2024](https://arxiv.org/html/2607.15592#bib.bib137)\)mainly addresses the second challenge by expert\-style modality selection and adaptively emphasizing different modalities under different relations\. Although these methods have improved multimodal entity ranking, their objectives remain fundamentally discriminative: they estimate compatibility scores over candidate entities, rather than explicitly modeling a generative refinement process for missing entity completion under multimodal uncertainty\.

Inspired by the recent success of diffusion models in complex conditional generation\(Nicholet al\.,[2021](https://arxiv.org/html/2607.15592#bib.bib168); Sahariaet al\.,[2022](https://arxiv.org/html/2607.15592#bib.bib169); Rombachet al\.,[2022](https://arxiv.org/html/2607.15592#bib.bib170)\), researchers have recently begun to explore diffusion\-based formulations for multimodal knowledge graph completion\. By progressively denoising a latent variable toward a target entity representation, diffusion provides a natural mechanism for iterative refinement under uncertainty, making it particularly well suited to multimodal entity completion\. Recent works such as DiffusionCom\(Huanget al\.,[2025](https://arxiv.org/html/2607.15592#bib.bib112)\)and FDM\(Longet al\.,[2024](https://arxiv.org/html/2607.15592#bib.bib159)\)have demonstrated the promise of diffusion for modeling complex relational structures in knowledge graphs\. However, existing diffusion\-based MKGC methods typically inject raw multimodal features directly into the denoising process, implicitly assuming that the original multimodal feature space is already suitable for diffusion\. This design leaves the denoiser to simultaneously handle the two challenges above, namely cross\-modal semantic heterogeneity and relation\-dependent multimodal cue selection, while also performing structure\-aware entity generation\. Prior diffusion\-based formulations effectively denoise under raw multimodal conditions, thereby entangling multimodal cue selection, semantic alignment, and entity generation in a single process\.

Therefore, we argue that effective diffusion for MKGC should be built on arelation\-dependent and semantically alignedlatent space rather than the raw heterogeneous feature space\. Figure[2](https://arxiv.org/html/2607.15592#S1.F2)contrasts this idea with prior diffusion\-based formulations\. Based on this intuition, we proposeMLLM\-GuidedDiffusionTransformer with Relation\-AdaptiveMixture\-of\-Experts\(M2GDT\), a new framework for MKGC built on analign\-then\-diffuseparadigm\. As illustrated in Figure[2](https://arxiv.org/html/2607.15592#S1.F2), M2GDT explicitly addresses the above two challenges in a decoupled manner\. First, it performsRelation\-Adaptive Semantic Routingwith a mixture\-of\-experts \(MoE\) module, which dynamically selects modality\-specific semantic transformation paths under relation conditioning and thus tackles relation\-dependent multimodal cue selection\. Next, the routed multimodal representations are aligned into a unified semantic space under the guidance of a Multimodal Large Language Model \(MLLM\), which serves as a semantic anchor with strong cross\-modal priors and alleviates semantic heterogeneity across modalities\. Finally, diffusion is performed in this aligned latent space with aKnowledgeGraphDiffusionTransformer \(KGDT\), allowing the denoiser to focus on graph\-conditioned entity generation rather than being jointly burdened by multimodal cue selection, alignment, and generation\. Extensive experiments on three benchmark datasets demonstrate that M2GDT consistently outperforms strong baselines, validating the effectiveness of the proposed paradigm\.

![Refer to caption](https://arxiv.org/html/2607.15592v1/figures/align_then_diffuse_paradigm.png)Figure 2\.Comparison between the proposed method and existing methods\. Existing methods directly denoise under raw heterogeneous multimodal conditions, which entangles semantic alignment with entity generation\. In contrast, M2GDT first aligns multimodal representations with an MLLM semantic anchor and then performs structure\-aware diffusion generation with decoupled supervision\.Our contributions are summarized as follows:

- •We proposeMLLM\-GuidedDiffusionTransformer with Relation\-AdaptiveMixture\-of\-Experts\(M2GDT\), a novel framework for Multimodal Knowledge Graph Completion built on analign\-then\-diffuseparadigm, which performs diffusion generation in a relation\-dependent and semantically aligned latent space rather than directly in raw multimodal feature spaces\.
- •We design aRelation\-Adaptive Semantic Routing Mixture of Experts\(RASR\-MoE\)module that explicitly addresses relation\-dependent multimodal cue selection by dynamically choosing modality\-specific semantic transformation paths under relation conditioning, producing cleaner multimodal representations for subsequent MLLM\-anchored alignment\. We further employ a lightweight auxiliary regularization term to encourage complementary expert behaviors\.
- •We develop an MLLM\-anchored alignment and diffusion generation framework, in which routed multimodal representations are first calibrated into a unified semantic space to alleviate cross\-modal semantic heterogeneity and then refined by the proposedKnowledgeGraphDiffusionTransformer \(KGDT\) for graph\-conditioned denoising\. Extensive experiments on three benchmark datasets validate the effectiveness of the proposed framework and demonstrate consistent improvements over strong baselines\.

## 2\.Methodology

![Refer to caption](https://arxiv.org/html/2607.15592v1/figures/architecture.png)Figure 3\.Overview of theM2GDTframework\.RASR\-MoEfirst performs relation\-adaptive semantic routing over heterogeneous multimodal inputs\. A frozen semantic anchor then aligns the routed features into calibrated condition embeddings, which are combined with timestep embeddings for diffusion denoising\. Starting from a noisy latent,KGDTiteratively predicts noise and refines the latent toward the target entity representation\. The right panels show the architectures ofKGDTandRASR\-MoE\.### 2\.1\.Problem Formulation

A Multimodal Knowledge Graph is denoted as

\(1\)𝒢=\(𝒱,ℛ,𝒯,ℳ\),\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{R\},\\mathcal\{T\},\\mathcal\{M\}\),where𝒱\\mathcal\{V\},ℛ\\mathcal\{R\},𝒯\\mathcal\{T\}, andℳ\\mathcal\{M\}denote the entity set, relation set, factual triples, and multimodal attributes, respectively\. Given an incomplete query\(h,r,?\)\(h,r,?\)or\(?,r,t\)\(?,r,t\), the goal of MKGC is to infer the missing entity so that the completed triple is both semantically plausible and structurally consistent\. Unlike conventional discriminative methods that directly rank candidate entities, diffusion\-based methods formulate MKGC as a conditional generation problem in the latent space\. Without loss of generality, we focus on tail prediction\(h,r,?\)\(h,r,?\), while the head prediction case is symmetric\. Given the observed entityhh, relationrr, and the associated multimodal information, the model generates the missing entity representation𝐳^0\\hat\{\\mathbf\{z\}\}\_\{0\}through a denoising diffusion process, which is then matched against candidate entity embeddings for final ranking\.

### 2\.2\.Relation\-Adaptive Semantic Routing Mixture\-of\-Experts

A key assumption of M2GDT is that diffusion should be conditioned on semantically calibrated multimodal representations rather than uniformly fused raw features\. In multimodal knowledge graphs, different relations often depend on different semantic information\. Some relations are mainly determined by structural context, while others rely more on textual or visual cues\. Therefore, using a fixed multimodal encoding path for all queries may introduce unnecessary semantic interference before alignment\.

To address this issue, we introduce aRelation\-Adaptive Semantic Routing Mixture\-of\-Experts\(RASR\-MoE\) module, which performs relation\-conditioned expert routing before anchor\-guided alignment\. Its goal is not generic multimodal fusion, but selecting relation\-dependent semantic transformation paths that produce cleaner representations for subsequent MLLM\-guided alignment\.

Given a query\(h,r,?\)\(h,r,?\), we first retrieve the raw structural, visual, and textual features of the observed entityhh:

\(2\)𝐱s\\displaystyle\\mathbf\{x\}\_\{s\}=EntEmb​\(h\)∈ℝB×ds,\\displaystyle=\\mathrm\{EntEmb\}\(h\)\\in\\mathbb\{R\}^\{B\\times d\_\{s\}\},\(3\)𝐱v\\displaystyle\\mathbf\{x\}\_\{v\}=ImgEmb​\(h\)∈ℝB×dv,\\displaystyle=\\mathrm\{ImgEmb\}\(h\)\\in\\mathbb\{R\}^\{B\\times d\_\{v\}\},\(4\)𝐱t\\displaystyle\\mathbf\{x\}\_\{t\}=TxtEmb​\(h\)∈ℝB×dt,\\displaystyle=\\mathrm\{TxtEmb\}\(h\)\\in\\mathbb\{R\}^\{B\\times d\_\{t\}\},whereBBis the batch size, and𝐫∈ℝB×dr\\mathbf\{r\}\\in\\mathbb\{R\}^\{B\\times d\_\{r\}\}denotes the relation embedding\.

For each modalityk∈\{s,v,t\}k\\in\\\{s,v,t\\\}, we define a bank ofMMexperts

\{Ek\(1\),Ek\(2\),…,Ek\(M\)\}\.\\\{\\mathrm\{E\}\_\{k\}^\{\(1\)\},\\mathrm\{E\}\_\{k\}^\{\(2\)\},\\dots,\\mathrm\{E\}\_\{k\}^\{\(M\)\}\\\}\.A relation\-conditioned router computes the expert weights:

\(5\)𝐠~k=softmax​\(MLPk​\(\[𝐱k;𝐫\]\)\+𝜹k\),𝜹k∼𝒩​\(𝟎,Σk\),\\displaystyle\\tilde\{\\mathbf\{g\}\}\_\{k\}=\\mathrm\{softmax\}\\big\(\\mathrm\{MLP\}\_\{k\}\(\[\\mathbf\{x\}\_\{k\};\\mathbf\{r\}\]\)\+\\bm\{\\delta\}\_\{k\}\\big\),\\qquad\\bm\{\\delta\}\_\{k\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\Sigma\_\{k\}\),whereΣk\\Sigma\_\{k\}is predicted from the current modality representation\. This perturbation improves routing robustness and avoids over\-confident expert selection\.

Each expert produces a relation\-aware transformation

\(6\)𝐮k\(m\)=Ek\(m\)​\(𝐱k,𝐫\),m=1,…,M,\\displaystyle\\mathbf\{u\}\_\{k\}^\{\(m\)\}=\\mathrm\{E\}\_\{k\}^\{\(m\)\}\(\\mathbf\{x\}\_\{k\},\\mathbf\{r\}\),\\qquad m=1,\\dots,M,and the routed modality representation is obtained by

\(7\)𝐞~k=∑m=1M𝐠~k\(m\)⊙𝐮k\(m\),k∈\{s,v,t\}\.\\displaystyle\\tilde\{\\mathbf\{e\}\}\_\{k\}=\\sum\_\{m=1\}^\{M\}\\tilde\{\\mathbf\{g\}\}\_\{k\}^\{\(m\)\}\\odot\\mathbf\{u\}\_\{k\}^\{\(m\)\},\\qquad k\\in\\\{s,v,t\\\}\.
The routed features are then fused into a multimodal entity representation\. Concretely, we first project the routed modality features into a shared latent space:

\(8\)𝐞k=𝐖k​𝐞~k\+𝐛k,k∈\{s,v,t\},\\displaystyle\\mathbf\{e\}\_\{k\}=\\mathbf\{W\}\_\{k\}\\tilde\{\\mathbf\{e\}\}\_\{k\}\+\\mathbf\{b\}\_\{k\},\\qquad k\\in\\\{s,v,t\\\},where𝐖k∈ℝdf×dk\\mathbf\{W\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{f\}\\times d\_\{k\}\}and𝐛k∈ℝdf\\mathbf\{b\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{f\}\}are learnable modality\-specific projection parameters\. The fused entity representation is then obtained by a weighted sum:

\(9\)𝐞f=𝐞s\+𝐞v\+𝐞t,\\displaystyle\\mathbf\{e\}\_\{f\}=\\mathbf\{e\}\_\{s\}\+\\mathbf\{e\}\_\{v\}\+\\mathbf\{e\}\_\{t\},
In this way, RASR\-MoE performs relation\-aware semantic routing before alignment, while the subsequent lightweight fusion aggregates the routed modality features into unified multimodal representations for thealign\-then\-diffuseprocess\. To preserve task\-relevant discriminative capability in the RASR\-MoE encoder, we further apply a supervision term to the structural, visual, textual, and fused branches\. Following the encoder design, these branches produce prediction scores over all candidate entities:

\(10\)𝐩s,𝐩v,𝐩t,𝐩f∈ℝB×N,\\displaystyle\\mathbf\{p\}\_\{s\},\\mathbf\{p\}\_\{v\},\\mathbf\{p\}\_\{t\},\\mathbf\{p\}\_\{f\}\\in\\mathbb\{R\}^\{B\\times N\},whereNNis the number of entities\. The corresponding auxiliary supervision loss is defined as

\(11\)ℒmoe=BCE​\(𝐩s,𝐲\)\+BCE​\(𝐩v,𝐲\)\+BCE​\(𝐩t,𝐲\)\+BCE​\(𝐩f,𝐲\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{moe\}\}=\\mathrm\{BCE\}\(\\mathbf\{p\}\_\{s\},\\mathbf\{y\}\)\+\\mathrm\{BCE\}\(\\mathbf\{p\}\_\{v\},\\mathbf\{y\}\)\+\\mathrm\{BCE\}\(\\mathbf\{p\}\_\{t\},\\mathbf\{y\}\)\+\\mathrm\{BCE\}\(\\mathbf\{p\}\_\{f\},\\mathbf\{y\}\),where𝐲∈ℝB×N\\mathbf\{y\}\\in\\mathbb\{R\}^\{B\\times N\}is the target label matrix\. Here,BCE​\(⋅,⋅\)\\mathrm\{BCE\}\(\\cdot,\\cdot\)denotes the binary cross\-entropy loss, which supervises each branch to assign higher scores to valid target entities and thus preserves task\-relevant discriminative signals in the RASR\-MoE encoder\.

To reduce redundancy among experts, we further introduce a lightweight auxiliary regularization term inspired by mutual\-information\-based expert disentanglement\. This term encourages different experts within the same modality to capture complementary relation\-dependent semantic transformations rather than collapsing to similar mappings:

\(12\)ℒreg=∑k∈\{s,v,t\}ℒclub\(k\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{reg\}\}=\\sum\_\{k\\in\\\{s,v,t\\\}\}\\mathcal\{L\}\_\{\\mathrm\{club\}\}^\{\(k\)\},whereℒclub\(k\)\\mathcal\{L\}\_\{\\mathrm\{club\}\}^\{\(k\)\}is a lightweight CLUB\-style regularizer for modalitykk\(Zhanget al\.,[2024](https://arxiv.org/html/2607.15592#bib.bib137)\)\. Since this term only serves as an auxiliary objective, we keep its weight small in the overall training loss\.

### 2\.3\.MLLM\-Anchored Representation Alignment

#### 2\.3\.1\.Frozen Anchor Construction

For the same batch of entities, we concatenate the raw structural, visual, and textual inputs:

\(13\)𝐱cat=\[𝐱s;𝐱v;𝐱t\]∈ℝB×\(ds\+dv\+dt\)\.\\displaystyle\\mathbf\{x\}\_\{\\mathrm\{cat\}\}=\[\\mathbf\{x\}\_\{s\};\\mathbf\{x\}\_\{v\};\\mathbf\{x\}\_\{t\}\]\\in\\mathbb\{R\}^\{B\\times\(d\_\{s\}\+d\_\{v\}\+d\_\{t\}\)\}\.A learnable projection maps them into the hidden dimension of the frozen Qwen3\-VL language backbone:

\(14\)𝐱anchor=Projin​\(𝐱cat\)∈ℝB×dq,\\displaystyle\\mathbf\{x\}\_\{\\mathrm\{anchor\}\}=\\mathrm\{Proj\}\_\{\\mathrm\{in\}\}\(\\mathbf\{x\}\_\{\\mathrm\{cat\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{q\}\},wheredq=4096d\_\{q\}=4096in our implementation\.

The projected features are treated as continuous input embeddings and fed into the frozen backbone:

\(15\)𝐡anchor=Qwenfrozen​\(𝐱anchor\)∈ℝB×dq\.\\displaystyle\\mathbf\{h\}^\{\\mathrm\{anchor\}\}=\\mathrm\{Qwen\}\_\{\\mathrm\{frozen\}\}\(\\mathbf\{x\}\_\{\\mathrm\{anchor\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{q\}\}\.All parameters of the Qwen backbone are frozen throughout training\.

#### 2\.3\.2\.Modality\-specific Anchor Outputs

The frozen hidden states are projected into modality\-specific spaces to obtain anchor features with the same output dimensions as the MoE encoder:

\(16\)𝐚s\\displaystyle\\mathbf\{a\}\_\{s\}=Projsa​\(𝐡anchor\)∈ℝB×ds,\\displaystyle=\\mathrm\{Proj\}^\{a\}\_\{s\}\(\\mathbf\{h\}^\{\\mathrm\{anchor\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{s\}\},\(17\)𝐚v\\displaystyle\\mathbf\{a\}\_\{v\}=Projva​\(𝐡anchor\)∈ℝB×dv,\\displaystyle=\\mathrm\{Proj\}^\{a\}\_\{v\}\(\\mathbf\{h\}^\{\\mathrm\{anchor\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{v\}\},\(18\)𝐚t\\displaystyle\\mathbf\{a\}\_\{t\}=Projta​\(𝐡anchor\)∈ℝB×dt,\\displaystyle=\\mathrm\{Proj\}^\{a\}\_\{t\}\(\\mathbf\{h\}^\{\\mathrm\{anchor\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{t\}\},\(19\)𝐚f\\displaystyle\\mathbf\{a\}\_\{f\}=Projfa​\(𝐡anchor\)∈ℝB×df\.\\displaystyle=\\mathrm\{Proj\}^\{a\}\_\{f\}\(\\mathbf\{h\}^\{\\mathrm\{anchor\}\}\)\\in\\mathbb\{R\}^\{B\\times d\_\{f\}\}\.Therefore, the MoE encoder outputs and the anchor outputs have matched tensor shapes and can be aligned directly\.

#### 2\.3\.3\.Feature\-level Alignment

We align the MoE output features𝐞s,𝐞v,𝐞t,𝐞f\\mathbf\{e\}\_\{s\},\\mathbf\{e\}\_\{v\},\\mathbf\{e\}\_\{t\},\\mathbf\{e\}\_\{f\}with the corresponding anchor features𝐚s,𝐚v,𝐚t,𝐚f\\mathbf\{a\}\_\{s\},\\mathbf\{a\}\_\{v\},\\mathbf\{a\}\_\{t\},\\mathbf\{a\}\_\{f\}\.

For the structural, visual, and textual branches, we use cosine\-similarity\-based alignment:

\(20\)ℒk=1−1B​∑n=1Bcos⁡\(𝐞k,n,𝐚k,n\),k∈\{s,v,t\},\\displaystyle\\mathcal\{L\}\_\{k\}=1\-\\frac\{1\}\{B\}\\sum\_\{n=1\}^\{B\}\\cos\\\!\\left\(\\mathbf\{e\}\_\{k,n\},\\mathbf\{a\}\_\{k,n\}\\right\),\\qquad k\\in\\\{s,v,t\\\},where𝐞k,n\\mathbf\{e\}\_\{k,n\}and𝐚k,n\\mathbf\{a\}\_\{k,n\}denote thenn\-th sample in the batch\.

For the fused branch, we use a temperature\-scaled KL matching loss:

\(21\)𝐩f,n\\displaystyle\\mathbf\{p\}\_\{f,n\}=softmax​\(𝐞f,n/T\),\\displaystyle=\\mathrm\{softmax\}\(\\mathbf\{e\}\_\{f,n\}/T\),\(22\)𝐪f,n\\displaystyle\\mathbf\{q\}\_\{f,n\}=softmax​\(𝐚f,n/T\),\\displaystyle=\\mathrm\{softmax\}\(\\mathbf\{a\}\_\{f,n\}/T\),\(23\)ℒf=1B​∑n=1BKL​\(𝐩f,n∥𝐪f,n\),\\displaystyle\\mathcal\{L\}\_\{f\}=\\frac\{1\}\{B\}\\sum\_\{n=1\}^\{B\}\\mathrm\{KL\}\\\!\\left\(\\mathbf\{p\}\_\{f,n\}\\,\\\|\\,\\mathbf\{q\}\_\{f,n\}\\right\),whereTTis the temperature\.

The total alignment loss is defined as

\(24\)ℒalign=λk​ℒk\+λf​ℒf\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{align\}\}=\\lambda\_\{k\}\\mathcal\{L\}\_\{k\}\+\\lambda\_\{f\}\\mathcal\{L\}\_\{f\}\.In the current implementation, the strongest alignment weights are assigned to the structural and visual branches, which are heuristically set to 1\.0, while the textual and fused branches are used as auxiliary regularizers\.

### 2\.4\.Knowledge Graph Diffusion Transformer

With the multimodal condition now aligned in a unified semantic space, M2GDT performs the*diffuse*stage by generating the missing entity embedding through a conditional denoising diffusion process\. We instantiate the denoiser as aKnowledgeGraphDiffusionTransformer \(KGDT\), whose architecture is shown in Figure[3](https://arxiv.org/html/2607.15592#S2.F3)\. Compared with linear denoisers, a transformer\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.15592#bib.bib173)\)backbone is better suited for modeling long\-range relational dependencies and condition\-dependent interactions in knowledge graphs\.

![Refer to caption](https://arxiv.org/html/2607.15592v1/figures/anchor_alignment.png)Figure 4\.Illustration of the proposed anchor\-guided alignment mechanism\. The encoder outputs are aligned to frozen anchor references through feature\-level similarity losses, producing a semantically calibrated condition for diffusion\.#### 2\.4\.1\.Diffusion Condition under Joint Alignment

The anchor branch does not explicitly replace the encoder outputs with pre\-aligned features\. Instead, the alignment lossℒalign\\mathcal\{L\}\_\{\\mathrm\{align\}\}is optimized jointly with the diffusion objective during training, so that the encoder representations are progressively calibrated toward the frozen anchor references\. Therefore, the diffusion model is always conditioned on the current encoder features, while semantic alignment is enforced as a simultaneous training constraint\.

Concretely, at each training step, the routed modality representations produced by the RASR\-MoE encoder are

\(25\)𝐞s∈ℝB×df,𝐞v∈ℝB×df,𝐞t∈ℝB×df\.\\displaystyle\\mathbf\{e\}\_\{s\}\\in\\mathbb\{R\}^\{B\\times d\_\{f\}\},\\quad\\mathbf\{e\}\_\{v\}\\in\\mathbb\{R\}^\{B\\times d\_\{f\}\},\\quad\\mathbf\{e\}\_\{t\}\\in\\mathbb\{R\}^\{B\\times d\_\{f\}\}\.These features are then combined by element\-wise addition to form the entity\-side semantic condition:

\(26\)𝐞f=𝐞s\+𝐞v\+𝐞t∈ℝB×df,\\displaystyle\\mathbf\{e\}\_\{f\}=\\mathbf\{e\}\_\{s\}\+\\mathbf\{e\}\_\{v\}\+\\mathbf\{e\}\_\{t\}\\in\\mathbb\{R\}^\{B\\times d\_\{f\}\},
For each diffusion steptt, a learnable time projector maps the timestep index to a time embedding:

\(27\)𝝉t=Projtime​\(t\),\\displaystyle\\bm\{\\tau\}\_\{t\}=\\mathrm\{Proj\}\_\{\\mathrm\{time\}\}\(t\),which is broadcast to all samples in the batch\. The final condition embedding used by the denoiser is then

\(28\)𝐜t=𝐞f\+𝝉t\.\\displaystyle\\mathbf\{c\}\_\{t\}=\\mathbf\{e\}\_\{f\}\+\\bm\{\\tau\}\_\{t\}\.
In this way, semantic alignment and diffusion conditioning are learned jointly rather than sequentially: the denoiser always receives the current encoder condition, while the anchor\-guided loss continuously regularizes this condition toward a semantically coherent latent space throughout training\.

#### 2\.4\.2\.Forward Diffusion

Let𝐳0\\mathbf\{z\}\_\{0\}denote the target embedding of the missing entity\. Following DDPM\(Nichol and Dhariwal,[2021](https://arxiv.org/html/2607.15592#bib.bib150)\), the forward diffusion process gradually perturbs𝐳0\\mathbf\{z\}\_\{0\}with Gaussian noise:

\(29\)𝐳t=α¯t​𝐳0\+1−α¯t​ϵ,ϵ∼𝒩​\(𝟎,𝐈\),\\displaystyle\\mathbf\{z\}\_\{t\}=\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\mathbf\{z\}\_\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\bm\{\\epsilon\},\\qquad\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\),whereα¯t=∏i=1tαi\\bar\{\\alpha\}\_\{t\}=\\prod\_\{i=1\}^\{t\}\\alpha\_\{i\}andαt=1−βt\\alpha\_\{t\}=1\-\\beta\_\{t\}is determined by the variance schedule\{βt\}t=1T\\\{\\beta\_\{t\}\\\}\_\{t=1\}^\{T\}\.

Table 1\.MKGC performance comparison on MKG\-W, MKG\-Y and DB15K datasets\. The best results are marked in bold, the second\-best results are underlined\.
#### 2\.4\.3\.Conditional Reverse Denoising

The reverse denoising process is parameterized by KGDT:

\(30\)ϵ^=KGDT​\(𝐳t,𝐜t;Θ\),\\displaystyle\\hat\{\\bm\{\\epsilon\}\}=\\mathrm\{KGDT\}\(\\mathbf\{z\}\_\{t\},\\mathbf\{c\}\_\{t\};\\Theta\),whereΘ\\Thetadenotes the learnable parameters of the transformer\. As shown in Figure[3](https://arxiv.org/html/2607.15592#S2.F3), the first KGDT block injects𝐜t\\mathbf\{c\}\_\{t\}into the noisy latent through cross\-attention, and the subsequent transformer blocks iteratively refine the latent representation through stacked attention and feed\-forward transformations\. In this way, each denoising step is guided by semantically aligned multimodal information rather than by raw heterogeneous features\.

Given the predicted noise, the reverse transition follows the standard DDPM update:

\(31\)𝐳t−1=1αt​\(𝐳t−1−αt1−α¯t​ϵ^\)\+σt​𝜼,𝜼∼𝒩​\(𝟎,𝐈\),\\displaystyle\\mathbf\{z\}\_\{t\-1\}=\\frac\{1\}\{\\sqrt\{\\alpha\_\{t\}\}\}\\left\(\\mathbf\{z\}\_\{t\}\-\\frac\{1\-\\alpha\_\{t\}\}\{\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\}\\hat\{\\bm\{\\epsilon\}\}\\right\)\+\\sigma\_\{t\}\\bm\{\\eta\},\\qquad\\bm\{\\eta\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\),whereσt\\sigma\_\{t\}is determined by the diffusion schedule\.

### 2\.5\.Training and Inference

#### 2\.5\.1\.Training Objective

The full M2GDT framework is trained with four objectives: an auxiliary supervision term for the RASR\-MoE, a lightweight auxiliary expert regularization term for the RASR\-MoE, an anchor\-guided semantic alignment objective, and a conditional diffusion denoising objective\.

First, the auxiliary supervision lossℒmoe\\mathcal\{L\}\_\{\\mathrm\{moe\}\}preserves task\-relevant discriminative signals in the RASR\-MoE encoder, while the auxiliary regularization termℒreg\\mathcal\{L\}\_\{\\mathrm\{reg\}\}encourages different experts to capture complementary relation\-dependent semantic transformations\.

Second, the alignment lossℒalign\\mathcal\{L\}\_\{\\mathrm\{align\}\}encourages the routed multimodal features to match the frozen anchor references in a semantically coherent latent space\.

Third, KGDT is trained with a standard denoising objective in the aligned latent space\. Given a clean target entity embedding𝐳0\\mathbf\{z\}\_\{0\}, a sampled timesteptt, and Gaussian noiseϵ∼𝒩​\(𝟎,𝐈\)\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), the diffusion loss is formulated as

\(32\)ℒdiff=𝔼𝐳0,t,ϵ​\[‖KGDT​\(𝐳t,𝐜t\)−ϵ‖22\]\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{diff\}\}=\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{0\},t,\\bm\{\\epsilon\}\}\\left\[\\left\\\|\\mathrm\{KGDT\}\(\\mathbf\{z\}\_\{t\},\\mathbf\{c\}\_\{t\}\)\-\\bm\{\\epsilon\}\\right\\\|\_\{2\}^\{2\}\\right\]\.
The overall training objective is

\(33\)ℒtotal=λ1​ℒmoe\+λ2​ℒreg\+λ3​ℒalign\+λ4​ℒdiff\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\\lambda\_\{1\}\\mathcal\{L\}\_\{\\mathrm\{moe\}\}\+\\lambda\_\{2\}\\mathcal\{L\}\_\{\\mathrm\{reg\}\}\+\\lambda\_\{3\}\\mathcal\{L\}\_\{\\mathrm\{align\}\}\+\\lambda\_\{4\}\\mathcal\{L\}\_\{\\mathrm\{diff\}\}\.Here,λ1\\lambda\_\{1\},λ2\\lambda\_\{2\},λ3\\lambda\_\{3\}, andλ4\\lambda\_\{4\}control the relative importance of auxiliary supervision, expert regularization, semantic alignment, and diffusion denoising, respectively\. In the final model, we setλ1=1\.0\\lambda\_\{1\}=1\.0,λ2=0\.0001\\lambda\_\{2\}=0\.0001,λ3=0\.01\\lambda\_\{3\}=0\.01, andλ4=0\.1\\lambda\_\{4\}=0\.1\. This setting keeps the expert regularization lightweight, while balancing semantic alignment and diffusion denoising during joint optimization\.

#### 2\.5\.2\.Inference

During inference, M2GDT starts from Gaussian noise𝐳T∼𝒩​\(𝟎,𝐈\)\\mathbf\{z\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)and iteratively applies the reverse denoising process conditioned on the aligned multimodal representation\. AfterTTsteps, the model obtains the predicted entity embedding𝐳^0\\hat\{\\mathbf\{z\}\}\_\{0\}, which is then compared with candidate entity embeddings for ranking\-based completion\. In this way, semantic alignment is handled before generation, while KGDT focuses on structure\-aware denoising in the aligned latent space\.

## 3\.Experiments

### 3\.1\.Experimental Setup

Datasets\.We evaluate M2GDT on three widely used multimodal knowledge graph completion benchmarks: MKG\-W, MKG\-Y\(Xuet al\.,[2022](https://arxiv.org/html/2607.15592#bib.bib146)\), and DB15K\(Liuet al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib121)\)\. These datasets contain relational triples together with textual descriptions and visual attributes for entities, making them suitable for evaluating multimodal reasoning and completion\.

Task and Evaluation Metrics\.We follow the standard MKGC protocol, in which the task is to predict the missing head entity for\(?,r,t\)\(?,r,t\)or the missing tail entity for\(h,r,?\)\(h,r,?\)\. For each query, all entities in the dataset are considered as candidates and ranked according to the predicted score\. We report Mean Reciprocal Rank \(MRR\), Hits@1, Hits@3, and Hits@10, for which higher values indicate better performance\.

Baselines\.We compare M2GDT with 19 representative baselines, including 6 unimodal KGC methods and 12 multimodal KGC methods\. The unimodal baselines include TransE\(Bordeset al\.,[2013](https://arxiv.org/html/2607.15592#bib.bib117)\), DistMult\(Zhang and Zhang,[2022a](https://arxiv.org/html/2607.15592#bib.bib118)\), ComplEx\(Trouillonet al\.,[2016](https://arxiv.org/html/2607.15592#bib.bib153)\), RotatE\(Sunet al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib147)\), GC\-OTE\(Tanget al\.,[2020](https://arxiv.org/html/2607.15592#bib.bib172)\), and TuckER\(Balazevicet al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib124)\)\. The multimodal baselines include IKRL\(Xieet al\.,[2017](https://arxiv.org/html/2607.15592#bib.bib123)\), TBKGC\(Mousselly\-Sergiehet al\.,[2018](https://arxiv.org/html/2607.15592#bib.bib141)\), TransAE\(Wanget al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib155)\), MMKRL\(Chenet al\.,[2025](https://arxiv.org/html/2607.15592#bib.bib157)\), RSME\(Wanget al\.,[2021](https://arxiv.org/html/2607.15592#bib.bib158)\), VBKGC\(Zhang and Zhang,[2022b](https://arxiv.org/html/2607.15592#bib.bib156)\), OTKGE\(Caoet al\.,[2022](https://arxiv.org/html/2607.15592#bib.bib152)\), IMF\(Liet al\.,[2023b](https://arxiv.org/html/2607.15592#bib.bib133)\), AdaMF\-MAT\(Zhanget al\.,[2024](https://arxiv.org/html/2607.15592#bib.bib137)\), VISTA\(Leeet al\.,[2023](https://arxiv.org/html/2607.15592#bib.bib126)\), MyGo\(Zhanget al\.,[2025](https://arxiv.org/html/2607.15592#bib.bib165)\), MoMoK\(Zhanget al\.,[2024](https://arxiv.org/html/2607.15592#bib.bib137)\), and MCKGC\(Gaoet al\.,[2025](https://arxiv.org/html/2607.15592#bib.bib125)\)\.

Implementation Details\.Our model is implemented in PyTorch 2\.6\.0 and trained on a single NVIDIA V100 GPU with 32GB memory\. We use AdamW with an initial learning rate of1×10−41\\times 10^\{\-4\}and a weight decay of5×10−55\\times 10^\{\-5\}\. The batch size is set to 128 and the maximum number of training epochs is 3000\. Textual features are initialized with BERT\-base\(Devlinet al\.,[2019](https://arxiv.org/html/2607.15592#bib.bib106)\), visual features are extracted by ResNet\-50\(Heet al\.,[2016](https://arxiv.org/html/2607.15592#bib.bib104)\)pretrained on ImageNet\(Russakovskyet al\.,[2015](https://arxiv.org/html/2607.15592#bib.bib105)\), and structural embeddings are initialized from TuckER\. The number of KGDT blocks is set to 3, and the number of diffusion steps is set to 100\.

### 3\.2\.Comparison with State\-of\-the\-art

![Refer to caption](https://arxiv.org/html/2607.15592v1/figures/sensitivity.png)Figure 5\.Sensitivity analysis of M2GDT with respect to four hyperparameters on the MKG\-W dataset: \(a\) diffusion steps, \(b\) number of Transformer blocks, \(c\) MLLM alignment weight, and \(d\) diffusion loss weight\.Table[1](https://arxiv.org/html/2607.15592#S2.T1)reports the main results on MKG\-W, MKG\-Y, and DB15K\. Overall, M2GDT achieves the best MRR on all three datasets and also obtains the strongest Hits@1 and Hits@3 in nearly all cases, indicating clear improvements in top\-ranked prediction quality\. Although M2GDT is not the best method on MKG\-Y Hits@10, it remains competitive on this metric while outperforming prior methods on the more discriminative ranking measures\. These results show that M2GDT improves not only overall ranking performance, but more importantly the precision of the highest\-confidence predictions, which is particularly critical for entity completion\.

Compared with unimodal KGC baselines, M2GDT delivers substantial gains by incorporating multimodal cues into a relation\-dependent and semantically aligned generative framework\. For example, relative to RotatE, one of the strongest unimodal baselines, M2GDT improves MRR from 33\.67 to 37\.77 on MKG\-W, from 34\.95 to 39\.25 on MKG\-Y, and from 29\.28 to 40\.54 on DB15K\. This consistent margin suggests that purely structural reasoning is insufficient for multimodal entity completion, especially when textual and visual cues provide complementary semantics that cannot be captured by structure\-only models\.

Compared with strong multimodal baselines, M2GDT also shows clear and consistent advantages\. On MKG\-W, M2GDT improves MRR from 36\.88 to 37\.77 over the strongest previous result, while also achieving the best Hits@1 and Hits@3\. On MKG\-Y, M2GDT raises MRR from 38\.44 to 39\.25 over the best prior baseline, and further achieves the strongest Hits@1 and Hits@3\. On DB15K, M2GDT improves MRR from 39\.79 to 40\.54 and achieves the best performance on all reported metrics\. These gains are particularly meaningful because the compared baselines, including MyGo, MoMoK, and MCKGC, are already strong multimodal methods with competitive fusion or representation mechanisms\.

More importantly, the empirical advantage of M2GDT is consistent with our method design\. Relative to discriminative methods such as MoMoK, which mainly improve multimodal reasoning through relation\-aware expert selection, M2GDT further introduces relation\-adaptive semantic routing before alignment and performs diffusion\-based generation in an MLLM\-calibrated latent space\. This design allows the model to first select relation\-dependent multimodal cues, then reduce cross\-modal semantic inconsistency, and finally focus on structure\-aware entity generation\. Therefore, the gain of M2GDT does not only come from using more multimodal signals, but also from the coordinated effect ofrelation\-adaptive routing,semantic alignment, andaligned\-space diffusion generation\.

Table 2\.Ablation study results on the MKG\-W dataset\.
### 3\.3\.Ablation Study

To examine the contribution of each component in M2GDT, we conduct ablation experiments on the MKG\-W dataset, as reported in Table[2](https://arxiv.org/html/2607.15592#S3.T2)\. The ablated variants include removing one modality branch at a time \(w/o Text Branch,w/o Image Branch, andw/o Structure Branch\), removing the semantic anchor branch \(w/o Semantic Anchor\), removing the proposed relation\-adaptive semantic routing module \(w/o RASR\-MoE\), replacing KGDT with a weaker denoising backbone \(w/o KGDT\), and further replacing KGDT with a simple linear denoiser \(w/o KGDT \(Linear Denoiser\)\)\.

Overall, all ablated variants perform worse than the full M2GDT, confirming that the effectiveness of the framework comes from the coordinated interaction among relation\-adaptive semantic routing, MLLM\-guided alignment, and diffusion\-based generation\. Removing any modality branch consistently degrades performance, which shows that structural, textual, and visual signals provide complementary cues for multimodal entity completion\. Among them, removing the structure branch causes the largest drop \(MRR 37\.77→\\rightarrow35\.12\), indicating that graph structure remains the dominant source of supervision, while textual and visual modalities provide important complementary semantics\.

The most severe degradation is observed inw/o RASR\-MoE, where MRR drops to 30\.88\. This result verifies the importance of relation\-dependent multimodal cue selection, since removing semantic routing substantially weakens the model’s ability to select relation\-relevant cues before generation\. Thew/o Semantic Anchorvariant further reduces MRR to 33\.45, showing that anchor\-guided semantic calibration is critical for alleviating cross\-modal semantic heterogeneity and building a coherent diffusion condition\.

The denoising backbone also matters\. Replacing KGDT with a weaker denoising backbone reduces MRR to 34\.88, while a simple linear denoiser yields 36\.68\. Although the linear variant remains competitive, both still underperform the full M2GDT, indicating that semantically calibrated conditions alone are not sufficient and that the Transformer backbone is more effective for modeling condition\-dependent interactions and complex relational dependencies during iterative denoising\.

### 3\.4\.Hyperparameter Sensitivity Analysis

Figure[5](https://arxiv.org/html/2607.15592#S3.F5)analyzes the sensitivity of M2GDT to four important hyperparameters on MKG\-W\. Overall, all four curves exhibit clear optima, indicating that the model is stable within a reasonable parameter range rather than being overly sensitive to a single setting\.

Diffusion Steps\.As shown in Figure[5](https://arxiv.org/html/2607.15592#S3.F5)\(a\), performance improves substantially when the number of diffusion steps increases from 10 to 100, but saturates or slightly declines beyond that point\. This suggests that too few steps are insufficient for iterative refinement, whereas excessively many steps bring limited gains while increasing computational cost\. So we set the number of diffusion steps to 100\.

Transformer Blocks\.Figure[5](https://arxiv.org/html/2607.15592#S3.F5)\(b\) shows that increasing the depth of KGDT improves performance up to 3 blocks, after which the gains become marginal\. This indicates that a shallow Transformer is insufficient to capture relational dependencies, while deeper architectures provide diminishing returns\. We thus use 3 KGDT blocks in the final model\.

MLLM Alignment Weight\.Figure[5](https://arxiv.org/html/2607.15592#S3.F5)\(c\) shows a clear peak atλ2=0\.01\\lambda\_\{2\}=0\.01\. When the alignment weight is too small, the semantic anchor is not strong enough to effectively calibrate heterogeneous modalities; when it is too large, the alignment objective over\-constrains the latent space and reduces the flexibility of diffusion generation\. This result supports our view that the MLLM should serve as a semantic anchor rather than a dominant supervisor\.

Diffusion Loss Weight\.As shown in Figure[5](https://arxiv.org/html/2607.15592#S3.F5)\(d\), the best performance is achieved atλ3=0\.1\\lambda\_\{3\}=0\.1\. A too\-small diffusion weight weakens the generative training signal, whereas a too\-large value makes denoising dominate the overall objective and suppresses alignment\. This confirms that M2GDT requires a balanced interaction between semantic alignment and conditional generation\.

![Refer to caption](https://arxiv.org/html/2607.15592v1/similarity.png)Figure 6\.Evolution of cosine similarity for text, vision, structure, and average cross\-modal alignment during training on the MKG\-W dataset\.
### 3\.5\.Analysis and Generalization

Table 3\.Performance comparison using different MLLMs as semantic anchors on MKG\-W dataset\.To evaluate the generalization and theoretical rationality of ouralign\-then\-diffuseparadigm, we analyze its core decoupling design with quantitative similarity metrics and verify compatibility with alternative MLLMs\. This paradigm’s core advantage is decoupling semantic alignment from structured generation: heterogeneous multimodal features are calibrated into a unified MLLM\-guided semantic space, while the diffusion model preserves KG structural specificity, avoiding feature homogenization via moderate cross\-modal alignment\.

Quantitative validation via cosine similarity \(see Figure[6](https://arxiv.org/html/2607.15592#S3.F6)\) shows that modal similarities rise steadily with minor training fluctuations and converge at 0\.52∼\\sim0\.58 after 1800 epochs, with text \(0\.580\.58\)\>\>structure \(0\.550\.55\)\>\>vision \(0\.520\.52\) and an average of0\.550\.55\. This ranking reflects intrinsic modal differences with MLLM semantics, and the moderate interval balances semantic consistency and structural specificity—avoiding performance loss from over\-alignment \(\>0\.60\>0\.60\) or insufficient alignment \(<0\.50<0\.50\), which underpins our model’s generalization ability\.

Generalization experiments \(Table[3](https://arxiv.org/html/2607.15592#S3.T3)\) show replacing Qwen3\-VL\-7B with CLIP\-ViT\-L/14\(Radfordet al\.,[2021](https://arxiv.org/html/2607.15592#bib.bib108)\)achieves 36\.85% MRR \(0\.92% drop\), competitive with SOTA MCKGC \(36\.88% MRR\)\. This minimal degradation is consistent with Figure[6](https://arxiv.org/html/2607.15592#S3.F6), where cross\-modal similarity fluctuates only by±0\.01\\pm 0\.01across MLLMs, confirming stable alignment independent of MLLM architecture\. Performance gaps across MLLMs derive from their understanding ability: CLIP lacks strong language understanding compared with Qwen3\-VL, while even BLIP\-2\(Liet al\.,[2023a](https://arxiv.org/html/2607.15592#bib.bib109)\)maintains superiority \(36\.42% MRR\)\.

These results confirm thealign\-then\-diffuseparadigm is a robust, transferable foundation for MKGC, adaptable to diverse multimodal backbones for practical deployment\. Our core contribution is the decoupling design unifying semantic alignment and structural generation, rather than reliance on a specific MLLM\.

## 4\.Conclusion

In this paper, we proposedMLLM\-Guided Diffusion Transformer with Relation\-Adaptive Mixture\-of\-Experts, analign\-then\-diffuseframework forMultimodal Knowledge Graph Completion\. The framework addresses two key challenges in Multimodal Knowledge Graph Completion: cross\-modal semantic heterogeneity and relation\-dependent multimodal cue selection\. We first introduce aRelation\-Adaptive Semantic Routing Mixture\-of\-Expertsmodule to perform relation\-dependent semantic routing over multimodal inputs, then use a frozenMultimodal Large Language Modelas a semantic anchor to align the routed representations into a unified latent space, and finally perform graph\-conditioned diffusion generation with aKnowledge Graph Diffusion Transformer\. In this way, the framework decouples multimodal cue selection, semantic calibration, and structure\-aware generation into a coherent pipeline\. Extensive experiments validate the effectiveness of the framework\.

## References

- I\. Balazevic, C\. Allen, and T\. Hospedales \(2019\)TuckER: tensor factorization for knowledge graph completion\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 5184–5193\.External Links:[Link](http://dx.doi.org/10.18653/v1/D19-1522),[Document](https://dx.doi.org/10.18653/v1/d19-1522)Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- A\. Bordes, N\. Usunier, A\. Garcia\-Durán, J\. Weston, and O\. Yakhnenko \(2013\)Translating embeddings for modeling multi\-relational data\.InProceedings of the 27th International Conference on Neural Information Processing Systems \- Volume 2,NIPS’13,Red Hook, NY, USA,pp\. 2787–2795\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Z\. Cao, Q\. Xu, Z\. Yang, Y\. He, X\. Cao, and Q\. Huang \(2022\)OTKGE: multi\-modal knowledge graph embeddings via optimal transport\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 39090–39102\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/ffdb280e7c7b4c4af30e04daf5a84b98-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Z\. Chen, Y\. Fang, Y\. Zhang, L\. Guo, J\. Chen, J\. Z\. Pan, H\. Chen, and W\. Zhang \(2025\)Noise\-powered multi\-modal knowledge graph representation framework\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 141–155\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p4.2)\.
- Y\. Gao, F\. Zhang, Z\. Zhang, X\. Min, and F\. Zhuang \(2025\)Mixed\-curvature multi\-modal knowledge graph completion\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 11699–11707\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p4.2)\.
- W\. Huang, M\. Liang, P\. Li, X\. Hou, Y\. Li, J\. Du, Z\. Xue, and Z\. Guan \(2025\)DiffusionCom: structure\-aware multimodal diffusion model for multimodal knowledge graph completion\.arXiv preprint arXiv:2504\.06543\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p4.1)\.
- J\. Lee, C\. Chung, H\. Lee, S\. Jo, and J\. Whang \(2023\)VISTA: visual\-textual knowledge graph representation learning\.InFindings of the association for computational linguistics: EMNLP 2023,pp\. 7314–7328\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023a\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational conference on machine learning,pp\. 19730–19742\.Cited by:[§3\.5](https://arxiv.org/html/2607.15592#S3.SS5.p3.1)\.
- X\. Li, X\. Zhao, J\. Xu, Y\. Zhang, and C\. Xing \(2023b\)IMF: interactive multimodal fusion model for link prediction\.InProceedings of the ACM web conference 2023,pp\. 2572–2580\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Y\. Liu, H\. Li, A\. Garcia\-Duran, M\. Niepert, D\. Onoro\-Rubio, and D\. S\. Rosenblum \(2019\)MMKG: multi\-modal knowledge graphs\.InEuropean Semantic Web Conference,pp\. 459–474\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p1.1)\.
- X\. Long, L\. Zhuang, A\. Li, H\. Li, and S\. Wang \(2024\)Fact embedding through diffusion model for knowledge graph completion\.InProceedings of the ACM Web Conference 2024,pp\. 2020–2029\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p4.1)\.
- H\. Mousselly\-Sergieh, T\. Botschen, I\. Gurevych, and S\. Roth \(2018\)A multimodal translation\-based approach for knowledge graph representation learning\.InProceedings of the seventh joint conference on lexical and computational semantics,pp\. 225–234\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- A\. Nichol, P\. Dhariwal, A\. Ramesh, P\. Shyam, P\. Mishkin, B\. McGrew, I\. Sutskever, and M\. Chen \(2021\)Glide: towards photorealistic image generation and editing with text\-guided diffusion models\.arXiv preprint arXiv:2112\.10741\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p4.1)\.
- A\. Q\. Nichol and P\. Dhariwal \(2021\)Improved denoising diffusion probabilistic models\.InInternational conference on machine learning,pp\. 8162–8171\.Cited by:[§2\.4\.2](https://arxiv.org/html/2607.15592#S2.SS4.SSS2.p1.2)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§3\.5](https://arxiv.org/html/2607.15592#S3.SS5.p3.1)\.
- R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer \(2022\)High\-resolution image synthesis with latent diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10684–10695\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p4.1)\.
- O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein,et al\.\(2015\)Imagenet large scale visual recognition challenge\.International journal of computer vision115\(3\),pp\. 211–252\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p4.2)\.
- C\. Saharia, W\. Chan, S\. Saxena, L\. Lit, J\. Whang, E\. Denton, S\. K\. S\. Ghasemipour, B\. K\. Ayan, S\. S\. Mahdavi, R\. Gontijo\-Lopes, T\. Salimans, J\. Ho, D\. J\. Fleet, and M\. Norouzi \(2022\)Photorealistic text\-to\-image diffusion models with deep language understanding\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p4.1)\.
- Z\. Sun, Z\. Deng, J\. Nie, and J\. Tang \(2019\)RotatE: knowledge graph embedding by relational rotation in complex space\.CoRRabs/1902\.10197\.External Links:[Link](http://arxiv.org/abs/1902.10197),1902\.10197Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Y\. Tang, J\. Huang, G\. Wang, X\. He, and B\. Zhou \(2020\)Orthogonal relation transforms with graph context modeling for knowledge graph embedding\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 2713–2722\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- T\. Trouillon, J\. Welbl, S\. Riedel, É\. Gaussier, and G\. Bouchard \(2016\)Complex embeddings for simple link prediction\.InInternational conference on machine learning,pp\. 2071–2080\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2\.4](https://arxiv.org/html/2607.15592#S2.SS4.p1.1)\.
- M\. Wang, S\. Wang, H\. Yang, Z\. Zhang, X\. Chen, and G\. Qi \(2021\)Is visual context really helpful for knowledge graph? a representation learning perspective\.InProceedings of the 29th ACM International Conference on Multimedia,MM ’21,New York, NY, USA,pp\. 2735–2743\.External Links:ISBN 9781450386517,[Link](https://doi.org/10.1145/3474085.3475470),[Document](https://dx.doi.org/10.1145/3474085.3475470)Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Z\. Wang, L\. Li, Q\. Li, and D\. Zeng \(2019\)Multimodal data enhanced representation learning for knowledge graphs\.In2019 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- R\. Xie, Z\. Liu, H\. Luan, and M\. Sun \(2017\)Image\-embodied knowledge representation learning\.InProceedings of the 26th International Joint Conference on Artificial Intelligence,IJCAI’17,pp\. 3140–3146\.External Links:ISBN 9780999241103Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- D\. Xu, T\. Xu, S\. Wu, J\. Zhou, and E\. Chen \(2022\)Relation\-enhanced negative sampling for multimodal knowledge graph completion\.InProceedings of the 30th ACM international conference on multimedia,pp\. 3857–3866\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p1.1)\.
- Y\. Zhang, Z\. Chen, L\. Guo, Y\. Xu, B\. Hu, Z\. Liu, W\. Zhang, and H\. Chen \(2025\)Tokenization, fusion, and augmentation: towards fine\-grained multi\-modal entity representation\.InAAAI,pp\. 13322–13330\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Y\. Zhang, Z\. Chen, L\. Liang, H\. Chen, and W\. Zhang \(2024\)Unleashing the power of imbalanced modality information for multi\-modal knowledge graph completion\.arXiv preprint arXiv:2402\.15444\.Cited by:[§1](https://arxiv.org/html/2607.15592#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.15592#S2.SS2.p8.2),[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Y\. Zhang and W\. Zhang \(2022a\)Knowledge graph completion with pre\-trained multimodal transformer and twins negative sampling\.arXiv preprint arXiv:2209\.07084\.Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.
- Y\. Zhang and W\. Zhang \(2022b\)Knowledge graph completion with pre\-trained multimodal transformer and twins negative sampling\.External Links:2209\.07084,[Link](https://arxiv.org/abs/2209.07084)Cited by:[§3\.1](https://arxiv.org/html/2607.15592#S3.SS1.p3.1)\.