multi-modal

TheBioCollection is a 52.6B-token pre-training corpus for biology that consolidates heterogeneous resources into a unified format for training large language models, and includes a matched evaluation suite. Training on this corpus more than doubles overall scores on biology benchmarks while preserving general language ability.

SenseTime released SenseNova-Vision, a 7B parameter model that unifies computer vision tasks as generation, with open weights, instruction corpus, benchmark, paper, and demo.

BRAID is a framework that formulates interleaved text-image-text reasoning as a unified Markov decision process, enabling joint optimization of textual and visual generation via reinforcement learning with a VLM judge providing dense turn-level feedback.

This paper proposes a proof-of-concept AI pipeline that uses multi-modal data (images and accident reports) to assess railway crossing safety, achieving a macro F1 score of 0.757 for risk classification and an RMSE of 0.078 for safety score estimation using a fine-tuned compact VLM.

NVIDIA releases Cosmos3-Edge, an omnimodal world foundation model that generates video, image, audio, and action commands from text, image, video, and action trajectory inputs, targeting Physical AI applications in robotics, autonomous driving, and smart spaces.

This paper introduces HealthAgentBench, a suite of 54 realistic healthcare tasks for evaluating frontier AI agents. It finds that even the best agent (Codex GPT-5.5) achieves only ~42% success, highlighting substantial room for improvement.

This paper introduces MMG-Pop, a unified benchmark for multi-modal graph-based social media popularity prediction, and proposes MMG-PopNet, a model that jointly models multimodal content and temporal social interactions. Experiments on Bluesky and Reddit datasets demonstrate superior performance and provide insights into cross-platform generalization and multi-task prediction.

This paper systematically compares seven state-of-the-art sensor fusion methods for multi-modal human activity recognition on the HARMES dataset, showing that Gated Multi-modal Fusion achieves the highest macro F1-score of 0.82, outperforming the baseline by 6 percentage points.

An open-source alternative to Google's NotebookLM offers free unlimited access, runs locally under MIT license, and includes features like podcast generation and multi-modal ingestion.

Microsoft open-sourced JARVIS, a system that uses a GPT controller to orchestrate hundreds of AI models from HuggingFace for multi-modal tasks.

OmniPath is a multi-modal agentic framework that combines OpenStreetMap network topology with aerial LiDAR data to audit wheelchair accessibility by analyzing physical barriers like slope and surface discontinuities at high resolution, validated against field surveys.

IV-CoT decomposes visual conditioning into structural and semantic cascades for improved structure-aware image generation, using training-only sketch supervision to guide structural queries. It achieves state-of-the-art results on GenEval and T2I-CompBench.

ChartWalker introduces a novel framework for cross-chart retrieval-augmented generation (RAG) using hierarchical knowledge graph construction and structure-aware sampling. It releases a challenging benchmark (ChartWalker-Bench) and an agentic baseline (ChartWalker-Agent), revealing significant performance gaps in current RAG paradigms.

Merve (@mervenoyann) shares day two findings of a pipeline using multiple small VLMs as judges for road sign detection, achieving map@50=0.8028 with only 1.3k examples. The thread compares model rejection rates and discusses dataset shrinking, super-specific prompts, and plans to generalize the library.

This research paper proposes a framework for fair token allocation and private data valuation in decentralized multi-modal agentic systems, using differentially private prototypes to balance privacy and utility while scheduling limited edge AI resources.

Mistral AI releases Mistral Medium 3.5, an open-source 128B dense model with 256k context, multimodal input, configurable reasoning, and agentic capabilities.

LLM Gateway Chat is a platform that provides access to multiple AI models for chat, image, video, and audio generation.

PermaVid introduces a multi-modal context memory that disentangles appearance and geometric structure to maintain long-term video consistency after editing operations, outperforming prior methods.

At PyTorch Conference Europe 2026, Mistral AI's Patrick von Platen explains why real-world AI interaction requires streaming architectures that process continuous input and produce continuous output, using Vox Real Time as a live transcription example.

This paper introduces a Multi-Modal Agent framework for power distribution defect detection, evaluating foundation models on perception, reasoning, and tool usage capabilities, with a new domain-specific dataset and benchmark.