cross-modal

The paper proposes a multimodal solution for audio sentiment polarity classification that integrates audio and multilingual text transcripts via cross-modal transformers, and uses knowledge distillation to enhance an audio-only model without computational overhead during inference.

This paper introduces LaViD, a framework that transfers semantic knowledge from a language-only LLM to a vision student model by generating multiple-choice questions as conceptual signatures, achieving superior fine-grained classification performance and robustness.

This paper reviews current multimodal LLM evaluation benchmarks and identifies key gaps such as temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention, arguing that existing isolated-task benchmarks fail to measure true cross-modal integration.

This paper proposes a cross-modal knowledge distillation framework that works without paired data by aligning feature and label distributions, offering theoretical guarantees and outperforming prior methods on multimodal benchmarks.

The article introduces SynIB, a scalable objective based on the information bottleneck principle that targets synergistic information in multimodal learning by penalizing confident predictions when a modality is masked, improving performance on tasks requiring cross-modal reasoning.

This paper identifies that failures in visual reasoning often stem from breakdowns in dynamic cross-modal coordination between visual and textual evidence during chain-of-thought generation. It introduces DyCo-RL, a reinforcement learning framework that rewards effective cross-modal coordination, leading to improved reasoning performance.

This paper proposes a query-based cross-modal projector that compresses visual tokens via cross-attention to improve Mamba-based multimodal LLMs, boosting both performance and throughput on vision-language benchmarks while eliminating the need for manual 2D scan order design.

JD released JoyAI-Echo, a long video generation model capable of 5-minute multi-shot video with cross-modal memory for character and voice consistency, native audio+video generation, and 7.5x speed improvement via DMD distillation.

This paper introduces StenCE, a pretraining framework that uses cross-modal contrastive learning between ECG and X-ray angiography representations to detect severe coronary stenosis from ECGs, achieving high performance and enabling early diagnosis even in asymptomatic patients.

TIGER is an inference-time framework that mitigates hallucinations in multimodal generation by extracting observation and claim graphs and assigning risk scores to repair unsupported facts. It reduces unsupported content across image-to-text, image+text-to-text, audio-to-text, and video-to-text tasks.

This paper introduces UniKE, the first benchmark for cross-modal knowledge editing in unified multimodal models (UMMs), revealing a significant modality gap where text edits achieve 92% efficacy but only 18.5% transfer to image generation. It proposes Reasoning-augmented Parameter Editing to improve cross-modal transfer, with gains up to 18.6 percentage points.

LoMo proposes a data curation method that reformulates single-modality prompts into interleaved multimodal sequences to improve cross-modal representation alignment in vision-language models, achieving consistent gains on multiple benchmarks.

Presents SceneAligner, a deep learning approach for floorplan localization that uses 3D scene reconstruction and cross-modal correspondence learning to work in real-world environments with limited data.

This paper systematically investigates cross-modal skill injection, where a domain-expert LLM is merged into a VLM to induce emergent multimodal capabilities. It evaluates different scenarios (instruction-following, cross-lingual, mathematical reasoning), merging methods (TA, DARE, etc.), and hyperparameters, finding that TA and DARE perform well except in mathematical reasoning.

LatentUMM introduces dual latent alignment to improve cross-modal consistency in unified multimodal models by aligning transformations and stabilizing latent dynamics.

AuralSAM2 integrates audio into SAM2 via an AuralFuser module that generates sparse and dense prompts from audio-visual features, enhancing cross-modal segmentation while maintaining interactive efficiency.

MNAFT (Modality Neuron-Aware Fine-Tuning) is a novel approach that selectively updates language-specific and language-agnostic neurons in multimodal large language models to improve image translation while preserving pre-trained knowledge. The method outperforms state-of-the-art image translation techniques including cascaded models and standard fine-tuning approaches.