multimodal-learning

Introduces AutoProteinEngine (AutoPE), an LLM-driven agent framework that enables biologists without deep learning expertise to perform multimodal AutoML for protein engineering via natural language, showing improvements over zero-shot and manual fine-tuning approaches.

This paper proposes PhenMol, a structure-preserving framework for phenotype-aware molecular representation learning that integrates cellular phenotype information while preserving chemical structure neighborhood organization, improving molecular property prediction and drug discovery tasks.

The paper proposes SentiLLM, a framework that uses semantic-aligned structural abstraction to distill non-verbal modalities into text-like tokens for multimodal sentiment analysis with LLMs. It introduces a dual-stream salience-context calibration mechanism and achieves superior performance on four datasets.

This paper proposes HCIG, a hierarchical cross-modal incongruity graph network for multimodal sarcasm and cyberbullying detection, achieving state-of-the-art performance on the MMSD and MultiBully datasets.

The paper proposes converting multimodal patient data (text, labs, vitals) into a single natural language sequence and fine-tuning LLMs for clinical prediction, achieving comparable or better performance than specialized fusion architectures across three tasks.

Proposes Parallel Quantum Feature Augmentation (PQFA), a hybrid quantum-classical framework that applies shallow variational quantum circuits to enhance fused multimodal features, outperforming classical baselines with fewer parameters.

This paper evaluates state-of-the-art tabular models as encoders in image-tabular multimodal learning, addressing the challenge of using In-Context Learning models that require labels for embedding training and test instances.

Proposes a fine-grained multimodal framework with a Binary Advantage-weighting Ranking Loss for automatic depression detection, achieving state-of-the-art results on D-vlog and LMVD datasets.

ViQ presents a visual quantization framework that balances semantic richness and detail preservation in discrete representations, enabling efficient multimodal training with native-resolution inputs by using text-aligned pre-training and proximal representation learning.

MOLAR proposes a noise-aware framework for learning multimodal molecular representations from noisy labels by separating clean-property inference from observed label noise, outperforming baselines on molecular benchmarks.

This paper presents an information-theoretic analysis of multimodal learning, revealing the need to capture sample-specific interactions, and proposes DMIL, a paradigm that explicitly models and learns from these interactions via variational decomposition and fine-tuning, achieving superior performance.

This paper introduces GLACIER, a multimodal student-teacher foundation model that integrates molecular graphs, SMILES strings, and physicochemical descriptors to predict molecular properties efficiently. It leverages Finsler geometry-aware fusion and knowledge distillation from larger teacher models (MiniMol, MolFormer) to achieve high performance with a lightweight architecture.

LongMoE proposes a unified framework that jointly addresses modality missingness and longitudinal dynamics in multimodal clinical learning, using context-aware imputation, attentional tokenization, trajectory-aware encoding, and sparse mixture-of-experts routing. Experiments on ADNI, OASIS-3, and MIMIC-IV demonstrate improved robustness under missing modalities while remaining competitive in full-modality settings.

This paper introduces a plug-in calibration module that adjusts multimodal representations before fusion, using cross-modal context to suppress misleading signals and emphasize reliable ones, improving performance on multiple benchmarks.

This paper proposes semantic motion anchors, natural-language abstractions of gesture motion for co-speech gesture retrieval and synthesis. The method discretizes 3D gestures into body-hand motion primitives and grounds them in transcripts, achieving significant improvements in text-to-gesture retrieval and user preference in generation.

Introduces Balanced Multimodal Label Reshaping (BMLR), a method that addresses modality imbalance in multimodal learning by reshaping the label space to equalize mapping difficulty across modalities, improving performance across various architectures.

The paper proposes Synheart Capacity, a theory-driven multimodal learning framework that models cognitive capacity dynamics from wearable cardiac and electrodermal signals, enabling continuous estimation of mental effort and stress states.

This paper proposes MODIAD, a framework for multimodal online distributed industrial anomaly detection, addressing resource constraints with a Multi-class Intelligent Scheduling problem and a Resource Efficient Class-Wise Low Rank Adaptation (REC-LoRA) strategy. Experiments on MVTec 3D-AD and Eyecandies datasets demonstrate superior performance and efficiency.

This paper introduces Checkup2Action, a multimodal dataset and benchmark for generating patient-oriented action cards from clinical check-up reports, addressing the interpretability gap for laypersons.

This paper proposes AnisoAlign, a framework that addresses the modality gap in multimodal models by applying anisotropic geometric correction to enable effective unpaired modality alignment.