Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Hugging Face Daily Papers Papers
multimodal-ai medical-ai llm vision-language-models knowledge-memory deployment model-agnostic

Summary

该论文提出一个模型无关的框架，让冻结的LLM和VLM在部署后通过技能、知识记忆和多模态知识库三种外部经验形式持续学习，在六项医学基准上最高提升34.2%的性能，且可跨模型迁移。

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:23 PM

Paper page - Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Source: https://huggingface.co/papers/2610.09146 Published on Oct 6

·

Submitted byhttps://huggingface.co/yexiaohe

yxon Oct 9

Abstract

Largelanguagemodels(LLMs)andvision-languagemodels(VLMs)areusuallyfrozenafterdeployment,sotheydonotlearnfromthecasestheysolve.Thisisespeciallyconcerninginmedicine,wherenewclinicalevidence,updatedguidelines,andnewtherapiescanchangeestablishedpractice.Fine-tuningcanupdatethemodel,butitrequiresaccesstomodelweightsandadditionaltraining.Parameter-freemethodsavoidtraining,buttheymayoverfitafixedvalidationset,lackreliabledomainknowledge,orlosevisualdetailsbysavingexperienceonlyastext.Toaddresstheselimitations,wepresentamodel-agnosticframeworkthatallowsfrozenLLMsandVLMstolearnfromdeploymentexperiencethroughthreeformsofexternalexpertise:aSkillthatguidesreasoningandtooluse,aKnowledgeMemorythatstoresreliablefactssupportedbyearliercasesortrustedexternalevidence,andaMultimodalKnowledgeBasethatkeepsvisualexamplesandguidesthemodeltorelateeachretrievedcasetothecurrentimage.Insteadofrelyingonafixedvalidationset,avalidationstrategykeepsanupdateonlyifithelpsonnewcaseswithoutdegradingperformanceonearlierones.Acrosssixbenchmarkscoveringclinicaldiagnosis,clinicalworkflows,medicalreasoning,andmedicalandnon-medicalvisualreasoning,andwithfouropen-weightandclosed-sourcebasemodels,ourframeworkimprovesperformanceduringonlinedeploymentbyupto34.2%overthebasemodelonmedicaltasks,generalizestounseencases,transferstoothermodelswithoutfurtheroptimization,andworksinnon-medicaldomains.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.09146

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.09146 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.09146 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.09146 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

CausalEvo: Causal Skill Self-Evolution for Health LLM Agents

arXiv cs.LG

CausalEvo: Causal Skill Self-Evolution for Health LLM Agents 提出了一种因果技能自进化方法，通过在固定模型权重和规则数量下对比不同技能集的效果来决定保留或替换技能，在 MedCalc-Bench-Verified 六项任务上达到 60.00% 准确率，优于 SkillOpt（55.83%），且训练 token 消耗减少 55.6%。

Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

Hugging Face Daily Papers

该论文提出 SCOPED（Skill-Decoupled On-Policy Expert Distillation），通过训练能力专属专家并以奖励引导反思提取技能，在在线蒸馏中将程序性知识内化到统一的视觉-语言模型中，从而避免跨能力梯度干扰，并在八个多模态基准上超越骨干模型和混合能力 RL。