Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

Papers with Code Trending Papers
medical-ai vision-language-model multi-modal open-source transparent clinical-ai state-of-the-art

Summary

Hulu-Med is a transparent medical vision-language model that unifies understanding across text, 2D/3D images, and video, achieving state-of-the-art performance on 30 benchmarks while being fully open-source.

Real-world clinical decision-making grapples with integrating information from diverse data modalities, including medical text, 2D/3D images, and video, leading to inefficiencies and potential diagnostic oversights. While generalist vision-language models (VLMs) offer promise, their medical development faces challenges of opaque pipelines, data scarcity, and architectural inflexibility. Here we present Hulu-Med, a transparent medical VLM that unifies understanding across all these modalities. Built upon a unified patch-based vision encoder and an LLM decoder, Hulu-Med was progressively trained on 16.7 million (M) samples to scale from 2D to 3D and video comprehension. The medical-aware token reduction enables efficient training, requiring only 4,000 to 40,000 GPU hours for 7B to 32B parameter variants. Extensive evaluation across 30 benchmarks exhibits state-of-the-art performance, surpassing leading open-source models and competing with proprietary systems in tasks spanning visual question-answering, medical report generation, and complex reasoning in multilingual and rare disease scenarios. By open-sourcing our complete pipeline, we establish that high-performance medical VLM can be achieved transparently, providing a foundational tool for accessible and impactful clinical AI. Code is released on https://github.com/ZJUI-AI4H/Hulu-Med{https://github.com/ZJUI-AI4H/Hulu-Med}.
Original Article
View Cached Full Text

Cached at: 07/04/26, 12:37 PM

Paper page - Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

Source: https://huggingface.co/papers/2510.08668 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Hulu-Med, a transparent medical vision-language model, integrates diverse data modalities and achieves state-of-the-art performance across various clinical tasks with efficient training.

Real-world clinical decision-making grapples with integrating information from diverse data modalities, including medical text, 2D/3D images, and video, leading to inefficiencies and potential diagnostic oversights. While generalistvision-language models(VLMs) offer promise, their medical development faces challenges of opaque pipelines, data scarcity, and architectural inflexibility. Here we present Hulu-Med, a transparent medical VLM that unifies understanding across all these modalities. Built upon aunified patch-based vision encoderand anLLM decoder, Hulu-Med was progressively trained on 16.7 million (M) samples to scale from 2D to 3D and video comprehension. The medical-aware token reduction enables efficient training, requiring only 4,000 to 40,000 GPU hours for 7B to 32B parameter variants. Extensive evaluation across 30 benchmarks exhibits state-of-the-art performance, surpassing leading open-source models and competing with proprietary systems in tasks spanning visual question-answering,medical report generation, andcomplex reasoninginmultilingualandrare disease scenarios. By open-sourcing our complete pipeline, we establish that high-performance medical VLM can be achieved transparently, providing a foundational tool for accessible and impactful clinical AI. Code is released on https://github.com/ZJUI-AI4H/Hulu-Med{https://github.com/ZJUI-AI4H/Hulu-Med}.

View arXiv pageView PDFGitHub842Add to collection

Get this paper in your agent:

hf papers read 2510\.08668

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper12

#### ZJU-AI4H/Hulu-Med-7B Image-Text-to-Text• 8B• UpdatedNov 27, 2025 • 6.49k • 104 #### ZJU-AI4H/Hulu-Med-4B Image-Text-to-Text• 5B• UpdatedNov 27, 2025 • 5.53k • 84 #### ZJU-AI4H/Hulu-Med-32B Image-Text-to-Text• 33B• UpdatedNov 27, 2025 • 1.29k • 58 #### ZJU-AI4H/Hulu-Med-14B Image-Text-to-Text• 15B• UpdatedNov 27, 2025 • 111 • 52 Browse 12 models citing this paper## Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2510.08668 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2510.08668 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Hugging Face Daily Papers

ClinFusion is a vision-centric multimodal large language model for holistic medical understanding that unifies 2D and 3D medical image analysis using a cascaded vision encoder. It achieves state-of-the-art results on 20 out of 24 benchmarks and outperforms proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks.

Aloe-Vision: Robust Vision-Language Models for Healthcare

arXiv cs.CL

Aloe-Vision introduces a family of open medical Vision-Language Models trained on a quality-filtered mixture of medical and general data, along with a new benchmark CareQA-Vision for reliable evaluation. The models demonstrate competitive performance while highlighting vulnerabilities to adversarial inputs.

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

Papers with Code Trending

VideoChat3 is a fully open, efficient, and generalist video-centric multimodal large language model that introduces Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for streaming video perception, along with scalable video data synthesis pipelines, achieving superior performance with only 4B parameters.