multimodal-learning

This paper introduces M2G-LLM, a framework that enhances Large Language Models with multimodal graph reasoning via Graph Neural Networks to improve clinical prediction tasks using electronic health records.

The paper introduces Lens, a training-free framework for multimodal representation learning that addresses semantic perspective misalignment, achieving significant performance improvements on MMEB datasets without parameter updates.

The paper introduces the Modality Discrepancy Transformer (MDT), a novel multimodal fusion framework that enhances cross-modal discrepancy modeling for recognizing ambivalence and hesitancy in clinical videos, outperforming baselines on the BAH dataset.

This paper introduces OmniVChat, a task for native audio-visual dialogue, and presents a data engine, benchmark, and reinforcement learning method to train and evaluate omni models, demonstrating improvements on synthesized and human-recorded data.

FLAT introduces a shared sequence of continuous tokens for images and text, enabling flexible-length representations for multimodal tasks like generation and retrieval, with strong performance on benchmarks.

This paper introduces a neuro-symbolic Hierarchical Planning Decoder (HPD) for anticipating human intentions from multimodal episodes. It demonstrates improved performance in goal inference and logic constraint satisfaction on a benchmark dataset.

The paper proposes CoMA-DiT, a bidirectional cross-modal Diffusion Transformer for latent augmentation in multimodal brain state decoding, which enhances performance in tasks like auditory attention decoding and emotion recognition by using paired modalities as mutual supervisory signals.

This paper introduces a contrastive modeling framework for multimodal in-context learning to improve reasoning path alignment, enhancing performance on tasks like visual question answering.

This paper critiques optimization-based balancing strategies for multimodal sentiment analysis, showing they fail due to conflating fitting speed with discriminative importance, and proposes a new research agenda focusing on held-out modality valuation.

This paper proposes RiVaT-Fuse, a reliability-calibrated variational tensor fusion framework for multimodal prediction under modality uncertainty, which estimates a sample-wise latent state to improve predictive performance and stability in image–metadata tasks.

Radixark shares a blog post about how Miles supports multimodal learning for AI models with a shared post-training design for vision-language models and diffusion models.

TC-Next is a multimodal deep learning model that leverages foundation model forecasts and satellite imagery for zero-shot tropical cyclone track and intensity forecasting, showing significant error reduction over conventional trackers.

This paper introduces MEL, a coordinate-preserving EEG tokenization framework for translating EEG to fMRI, addressing representation-interface mismatch and improving prediction over baselines through explicit modeling of hemodynamic latency and spectral-spatial dynamics.

MedTVL is a text-guided dual-pathway architecture for medical time series classification that synergizes temporal and visual modalities with textual semantics, demonstrating superiority in various learning settings.

CoVA-SFT is a large-scale dataset and benchmark designed to train multimodal language models in chain of visual abstractions, improving performance over baselines in visual reasoning tasks.

SpanCalib-VLM presents a hybrid system that combines a multimodal sequence tagger with a generative vision-language model to enhance hallucination span detection and calibration, achieving improved performance on the SHROOM-Visions task.

The paper identifies strong-modality collapse in multimodal learning where fusion degrades the dominant modality's performance, and proposes Inverted Asymmetric Fusion (IAF) to preserve it, improving over unimodal baselines.

SAGE introduces a CLIP-based framework for time series forecasting that combines temporal, textual, and visual semantic information to improve accuracy, achieving state-of-the-art performance on long-term benchmarks.

NeoTriFuse is a reliability-aware multimodal fusion framework for neonatal mortality risk prediction that explicitly models missing data as a reliability signal to dynamically modulate feature contributions, improving performance on incomplete clinical monitoring data.

The paper proposes structured evidence routing, a router–predictor–reviewer workflow for incident risk prediction from multimodal longitudinal electronic health records, achieving competitive performance with interpretable patient-specific evidence trails.