Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
Summary
This paper proposes a cross-modal knowledge distillation framework that works without paired data by aligning feature and label distributions, offering theoretical guarantees and outperforming prior methods on multimodal benchmarks.
View Cached Full Text
Cached at: 06/10/26, 06:15 AM
# Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm Source: [https://arxiv.org/abs/2606.10504](https://arxiv.org/abs/2606.10504) [View PDF](https://arxiv.org/pdf/2606.10504) > Abstract:Cross\-modal knowledge distillation \(CMKD\) studies how a \(large\) teacher model trained on one type of data \(e\.g\., images\) can guide a \(smaller\) student model building on another type of data \(e\.g\., text/audio\)\. Existing CMKD methods often require paired multi\-modal data with aligned semantics, but obtaining such paired data are often costly and impractical\. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable\. In particular, we establish a cross\-modal distributional relationship between teacher and student models, which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment\. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively\. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross\-modal knowledge distillation by aligning distributions rather than individual samples\. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work\. ## Submission history From: Khiem Tran Trong \[[view email](https://arxiv.org/show-email/9b266a36/2606.10504)\] **\[v1\]**Tue, 9 Jun 2026 07:29:12 UTC \(1,693 KB\)
Similar Articles
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
This paper proposes a rationale-guided knowledge distillation framework for cross-lingual stance detection, using chain-of-thought prompting from large language models to train a compact student model with dual-path distillation and contrastive learning.
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Switch-KD proposes a novel visual-switch knowledge distillation framework for efficiently compressing vision-language models by unifying multimodal knowledge transfer within a shared text-probability space. The method achieves 3.6-point average improvement across 10 multimodal benchmarks when distilling a 0.5B TinyLLaVA student from a 3B teacher model.
Making Knowledge Distillation Cheap Enough to Run at Scale
Multiverse Computing announces a paper on making LLM knowledge distillation cheaper via offline top-K logits and a fused chunked KL loss, cutting VRAM usage for distillation at scale.
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
This paper proposes a unified approach to interpret knowledge distillation in LLMs using game-theoretic interactions, discovering that distillation sparsifies interactions, and introduces a loss function CIP to improve performance.
The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models
This paper shows that knowledge distillation has asymmetric effects on bias in small language models: it improves context-following on unambiguous tasks but harms refusal calibration on ambiguous ones, and proposes PCCD, a protocol to diagnose such per-item harms that aggregate metrics miss.