multimodal-learning

This paper introduces Drift Variation autoencoder, which uses conditional posterior flow matching to unify generative and representation learning, achieving high performance in controlled multimodal benchmarks.

CAT-GS introduces a neural dynamics-based optimization controller that stabilizes multimodal learning by addressing modality imbalance and fusion interference, achieving improved accuracy on various benchmarks.

LAION-BVD is a massive open video dataset for multimodal pre-training, containing 80 million videos and achieving competitive performance on standard benchmarks, released to support open research in AI.

Giraffe introduces a novel mapping architecture that translates hidden text representations to visual embeddings using a single token per image, enabling efficient graphic design generation in multimodal large language models.

The paper proposes a multimodal prompt-learning framework to handle missing modalities in electronic health records for robust clinical prediction in intensive care units, introducing four prompt types to capture dependencies and interactions.

LAION-BVD is a large-scale open video dataset containing 10 million hours of video data for multimodal pre-training, with synthetic captions and competitive benchmark performance.

This paper introduces a dual-tower architecture that integrates EEG data with literature-informed environmental priors for affective-state classification, demonstrating improved accuracy over EEG alone while highlighting methodological advances rather than causal exposure-affect associations.

The paper introduces Libra, a decoupled vision-language architecture for multimodal large language models that enables both image-to-text understanding and text-to-image generation, demonstrating strong performance on benchmarks.

This paper introduces the Disjoint-Bridge Operator-Substitution Certificate (DBOSC) to certify the interchangeability of modality compilers in physical language representation and evaluates ordered execution in controlled elastoplastic systems, separating distinct capabilities in attribute access, response substitution, fusion closure, and ordered execution.

This paper demonstrates that training only the projector in multimodal large language models achieves strong performance on 3D tasks, avoids language model drift, and improves training efficiency compared to joint training methods.

The paper proposes MARCUS, a missing-aware region representation model that treats missing data as contextual urban signals for rent prediction, achieving state-of-the-art performance with significant error reduction on real-world datasets.

This systematic review examines the applications of large language models in mental health, covering innovations in areas like clinical conversational agents and multimodal learning, while highlighting ethical challenges and advocating for safe deployment frameworks.

This paper studies how vision-language models shift their reliance between image and text sources when one modality is degraded, revealing task-dependent behavior and introducing a new conflict benchmark for evaluation.

The author describes building a world model harness that coordinates research agents to maximize evaluation metrics, achieving a 25x score improvement in a multimodal masked reconstruction task using geospatial inputs.

The paper proposes LUNG-KGMM, a knowledge-guided multimodal framework for predicting lung cancer incidence by integrating EHR, radiology data, and clinical guidelines, demonstrating superior performance on the MIMIC dataset and real-world validation.

This paper introduces The Unwritten Benchmark, a new challenge to evaluate abstract perceptual reasoning in multimodal AI models, revealing a significant performance gap between humans and current models like GPT-4o and Gemini 2.5-Pro.

The paper introduces Internalized Visual Thinking (IVT), a post-training framework that trains multimodal models to predict future frame embeddings, enabling direct answer generation without synthesizing intermediate images, thereby reducing latency over 5x for proactive video reasoning.

NARU is a benchmark for evaluating narrative evolution and cultural reasoning in Japanese extreme long-form videos, constructed through a hierarchical annotation pipeline and native-speaker verification to address gaps in current benchmarks.

GAS introduces a generation-guided training framework that improves visual understanding in multimodal models by using auxiliary generation tasks with no inference overhead, via a decoupled mixture-of-transformers architecture.

This paper analyzes multimodal systems for the CHiME-9 MCoRec cocktail-party scenario, comparing design strategies such as audio-visual target speech separation, improved recognition, and LLM-based conversational grouping, finding that speech overlap alone does not explain performance differences.