Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI
Summary
This paper presents a model-agnostic framework for per-modality failure analysis in multimodal clinical AI, distinguishing loud vs silent failures when a modality is dropped. Validated on planted ground truth and applied to EchoJEPA and HuBERT-ECG embeddings for LVEF prediction, it shows that dropping echo nearly doubles error.
View Cached Full Text
Cached at: 08/04/26, 05:40 PM
Paper page - Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI
Source: https://huggingface.co/papers/2608.01462 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Multimodalclinicalmodelsareusuallyjudgedonaccuracywitheverymodalitypresent,butdeploymentremovesmodalities;anechocardiogramisoftenunavailablewhereanECGisroutine.Twoquestionsthenmatterbeyondthesizeoftheaccuracyloss:whichmodalitywasresponsible,andwhetherthemodelfailsloudlyorsilentlyoncethatmodalityisdropped.Thedistinctionisper-exampleandmodality-level,andisseparatefrompost-hocfeatureattribution(e.g.SHAP).Modelsarereplacedoften;theevaluationthatanswersthesequestionsisreused.Wepresentamodel-agnosticmodality-failureframework:givenNmodalityembeddings,anymask-awareprobe,andlabels,itreturnsaper-examplefailuretaxonomy,aper-modalitycomplementaritymatrixthatattributeserrortomodalities,andaloud-vs-silentdropoutprofileseparatingmonitorablefailuresfromthosethatpassunflaggedfarfromthedecisionboundary,usingonlydeployment-observablesignals.Wereleaseitasasmall,unit-testedharnessandvalidateitagainstplantedgroundtruth.Acrossseedsitrecoversthatplantedmodalitydominanceandcomplementarysubset,reportsper-modalityloud-vs-silentrates,andscalestoathree-modalitycomplementaritymatrix;becausetheplantedstructureisknownbyconstruction,thisvalidatesrecoveryofper-exampleattributionratherthanclinicalperformance.WetheninstantiatetheframeworkonfrozenEchoJEPAandHuBERT-ECGembeddingsforLVEFandtheEF<=40%HFrEFgateoverapairedMIMIC-IVcohort,whereontheheld-outtestsplit(n=245)droppingechonearlydoubleserror.Thenarrowecho-to-ECGoverlapthatboundscohortsizeisitselfadeploymentfindingforcardiacfoundationmodels.Allofourworkcanbefoundathttps://github.com/criticaldata/PRIMED-AI.
View arXiv pageView PDFGitHub5Add to collection
Get this paper in your agent:
hf papers read 2608\.01462
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.01462 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.01462 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.01462 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Ten Failure Modes That Define Multimodal AI Systems
The article catalogs ten documented failure modes in multimodal AI systems where models generate fluent answers that break correspondence with actual inputs, based on benchmark papers and research studies.
Large Language Models as Unified Multimodal Learners for Clinical Prediction
The paper proposes converting multimodal patient data (text, labs, vitals) into a single natural language sequence and fine-tuning LLMs for clinical prediction, achieving comparable or better performance than specialized fusion architectures across three tasks.
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
This paper demonstrates that post-training quantization can silently alter how large language models reason, even when task accuracy is preserved, through a taxonomy-based analysis of 30,000 chain-of-thought outputs across multiple models and benchmarks.
Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients
The paper proposes a multimodal prompt-learning framework to handle missing modalities in electronic health records for robust clinical prediction in intensive care units, introducing four prompt types to capture dependencies and interactions.
From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detection
This paper presents a multi-stage explainable framework that combines SHAP-based token attribution, theory-informed linguistic features, and LLaMA-3.1-70B-Instruct LLM reasoning to interpret transformer-based speech models for cognitive impairment detection, achieving strong clinical alignment and high usability scores.