MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

Hugging Face Daily Papers Papers
medical multimodal foundation-models data-curation clip literature-curation clinical-validation

Summary

MedPMC is an automated framework that transforms medical literature into high-fidelity multimodal data for foundation models, achieving significant improvements across multiple benchmarks and clinical settings.

Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Original Article
View Cached Full Text

Cached at: 07/13/26, 03:51 PM

Paper page - MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

Source: https://huggingface.co/papers/2607.07673 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Medicineisinherentlymultimodal,requiringclinicianstosynthesizeinformationacrossdiversedatastreams.Yetthedevelopmentofmultimodalfoundationmodelsisconstrainedbylimitedaccesstolarge-scale,high-qualityclinicaldata.AlthoughPubMedCentral(PMC)offersacomplementarysourceofexpert-authoredimage-textdata,existingPMC-derivedresourcesremainlimitedinfidelity,reproducibility,andclinicalvalidation.WeintroduceMedPMC,anautomated,continuouslyupdatableframeworkthattransformspermissivelylicensedliteratureintohigh-fidelityinfrastructureformedicalmultimodalmodels.Appliedto6.1millionPMCarticles,MedPMCcurated11millionmedicalimage-textpairs.Componentevaluationsshowedstrongperformanceforinitialscreening(F1=93.2),multi-panelfiguredetection(F1=96.5),figureseparation(mAP=89.8),captionseparationandalignment(F1=81.4;ROUGE-L=85.3),andmedicalfigureclassification(F1=96.5).Manualreviewbyfiveannotators,threewithmedicaltraining,found95.3%ofMedPMCimagesmedicallyrelevant,versus19.7%inapriorPMC-deriveddataset.Across26benchmarksspanning11specialties,aMedPMC-trainedCLIP-stylemodelimprovedaveragezero-shotAUCby7.1percentagepointsoverthestrongestarchitecture-matchedbiomedicalCLIPbaselinedespiteusingfewerthanhalfasmanyimage-textpairs.Asthevisionencoderinamultimodallargelanguagemodel,itimprovedmedicalvisualquestion-answeringby1.9and16.9percentagepointsacrosstwobenchmarks.In10,524YaleNewHavenHealthSystemdermatologyphotographs,itimprovedmorphology-to-imageretrievalRecall@5by11.7percentagepoints.Thesefindingsshowthathigh-fidelityliteraturecurationstrengthensmedicalmultimodalfoundationmodelsacrossbenchmarkandclinicalsettings.Wepubliclyreleasetheframework,corpus,benchmarks,andpretrainedmodels.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.07673

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.07673 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.07673 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.07673 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles