Protecting patient privacy in clinical foundation models: Technical and legal perspectives

# Protecting patient privacy in clinical foundation models: Technical and legal perspectives
Source: [https://arxiv.org/html/2608.07705](https://arxiv.org/html/2608.07705)
[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x1.png)Sana Tonekaboni](https://orcid.org/0000-0000-0000-0000) Massachusetts Institute of Technology \(MIT\), Cambridge, USA The Broad Institute of MIT and Harvard, Cambridge, USA Borealis AI, Toronto, Canada stonekab@mit\.edu[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x2.png)Lena Stempfle\*](https://orcid.org/0000-0000-0000-0000) Massachusetts Institute of Technology \(MIT\), Cambridge, USA The Broad Institute of MIT and Harvard, Cambridge, USA stempfle@mit\.edu [![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x3.png)Sasha Ronaghi](https://orcid.org/0000-0000-0000-0000)

Stanford University, California, USA sronaghi@stanford\.edu

[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x4.png)Corinna Coupette](https://orcid.org/0000-0000-0000-0000)
Aalto University, Finland
Max Planck Institute for Tax Law and Public Finance, Germany
The Stanford Center for Legal Informatics, Stanford Law School, USA
corinna\.coupette@aalto\.fi
[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x5.png)I\. Glenn Cohen](https://orcid.org/0000-0000-0000-0000)
Harvard Law School, Cambridge, USA
Petrie\-Flom Center for Health Law Policy, Biotechnology & Bioethics
igcohen@law\.harvard\.edu
[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x6.png)Emily Alsentzer](https://orcid.org/0000-0000-0000-0000)
Stanford University, California, USA
ealsentzer@stanford\.edu
[![[Uncaptioned image]](https://arxiv.org/html/2608.07705v1/x7.png)Marzyeh Ghassemi](https://orcid.org/0000-0000-0000-0000)
Massachusetts Institute of Technology \(MIT\), Cambridge, USA
mghassem@mit\.edu
###### Abstract

Clinical foundation models trained on large\-scale patient data are increasingly used for decision support, screening, and public health\. As deployment expands, privacy risk increasingly arises from model\-mediated leakage, yet its prevalence and severity remain poorly quantified\. Models can disclose sensitive training artifacts, enabling patient re\-identification in ways not captured by data\-handling controls alone\. Existing frameworks, including HIPAA and GDPR, offer limited guidance for such indirect threats\. We propose a practical framework for assessing privacy risk in clinical foundation models and illustrate realistic leakage scenarios across deployment settings, map them to legal regimes, and outline complementary technical and legal mitigations\. Our analysis provides a context\-aware risk assessment grounded in realistic usage to preserve the value of medical foundation models while rigorously safeguarding patient privacy\.

*Keywords*AI safety⋅\\cdotPrivacy⋅\\cdotClinical foundation models

## 1Introduction

Medical foundation models \(FMs\) are rapidly integrating into clinical decision\-making, triage, screening, and public\-health planning\(Guptaet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib1); McKinneyet al\.,[2020](https://arxiv.org/html/2608.07705#bib.bib2); Tomaševet al\.,[2019](https://arxiv.org/html/2608.07705#bib.bib3); Bhargavaet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib4)\)\. Pretrained on large patient cohorts, they can be adapted to data\-scarce settings\(Mooret al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib5); Thamet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib6)\)and used to support diagnosis, personalize risk prediction, and streamline clinical workflows with the potential to improve patient outcomes\(Heet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib7); Sunet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib8)\)\.

As FMs grow in scale and are adapted more broadly, privacy risk can arise from unintended data leakage, enabling re\-identification even when trained on de\-identified records\(Benitez and Malin,[2010](https://arxiv.org/html/2608.07705#bib.bib17)\)\. Importantly, the leakage pathways are highly context\-dependent: they vary with how a model is shared, where it is deployed, and how it is used\. Most clinical FMs are trained on de\-identified personal health data\(Guoet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib9); Sellergrenet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib10); Guoet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib11)\)and deployed through two main pathways: \(i\) local deployment, and \(ii\) public release \(Figure[1](https://arxiv.org/html/2608.07705#S2.F1)\)\. In local deployments within healthcare systems, foundation models may be fine\-tuned on institutional patient data and accessed by clinicians through secured on\-premises or enterprise installations\(Rencet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib12)\)\. By contrast, public releases, via web\-based interfaces \(including general\-domain tools used for health purposes\)\(Google Research,[2022](https://arxiv.org/html/2608.07705#bib.bib13); OpenAI,[2026](https://arxiv.org/html/2608.07705#bib.bib14)\), application programming interfaces \(APIs\), or downloadable model weights and specifications\(Hanet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib15); Xieet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib16)\), enable external developers, researchers, healthcare providers, and commercial actors to integrate models into downstream applications, build new tools, or further adapt them through fine\-tuning and subsequent deployment\.

FMs trained on personal health data expand the privacy threat surface because risk arises not only from access to the underlying records, but also from what can be inferred or reconstructed from model outputs\. In this study, we focus on the privacy risk of information leakage that stems from model fragility and weakness\(Carliniet al\.,[2019](https://arxiv.org/html/2608.07705#bib.bib18); Khandelwalet al\.,[2019](https://arxiv.org/html/2608.07705#bib.bib19); Benitez and Malin,[2010](https://arxiv.org/html/2608.07705#bib.bib17); Tirumalaet al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib21)\), rather than from explicitly sophisticated attacks such as social engineering, insider misuse, or targeted model exploitation\. These risks are orthogonal to familiar clinical\-environment threats such as phishing, credential compromise, and large\-scale data breaches \(as described in the U\.S\. Department of Health and Human Services breach portal\(U\.S\. Department of Health and Human Services, Office for Civil Rights,[2025](https://arxiv.org/html/2608.07705#bib.bib22)\)\), which in practice often provide a simpler and more reliable path to obtain sensitive information than model\-centric attacks\. However, leakage that occurs through model behaviour are not well addressed by existing regulatory frameworks such as HIPAA \(Health Insurance Portability and Accountability Act\)\(U\.S\. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation,[1996](https://arxiv.org/html/2608.07705#bib.bib23)\), the GDPR \(General Data Protection Regulation\)\(European Parliament and Council of the European Union,[2016](https://arxiv.org/html/2608.07705#bib.bib24)\), or commissioned policy reports\(Novelliet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib25); Barberá,[2025](https://arxiv.org/html/2608.07705#bib.bib26)\), because existing frameworks were primarily designed to govern static risks to the direct handling of data \(e\.g\., data collection, access, and disclosure\)\. As such, they tend to adopt a rigid, audit\-oriented perspective, rather than accounting for dynamic, model\-mediated leakage that can arise through learned parameters and generated outputs\.

In this work, we examine the spectrum of leakage mechanisms in medical foundation models, propose a practical framework to assess their privacy risk, situate these risks within current legal regimes, and outline complementary technical and legal mitigation strategies\.

First, we introduce a two\-dimensional framework for assessing privacy risk in medical foundation models, characterized by i\) the amount of prior information required to trigger data leakage, and ii\) the type and extent of information that may be revealed\. We find that risk occurs at various levels, from leaking pieces of linkable personal information to identifiable personal information, resembling categories recognized in privacy law and data protection regulation\. We note that leakage of isolated linkable personal information poses minimal risk to patient privacy, but can be serious because it increases the likelihood of re\-identification\. Leakage of identifiable personal information represents the highest level of risk, as it directly exposes sensitive patient data\.

Second, we situate these risks within existing legal frameworks for patient data protection, focusing on the regulatory landscape in the US and the EU, existing laws \(HIPAA, GDPR, EU Artificial Intelligence Act \(AIA\)\)\. We find that existing legal definitions of privacy protection are underspecified, and call for context\-aware assessments grounded in realistic use and leakage scenarios rather than abstract or purely formal criteria\(Holzenberger and Maxwell,[2025](https://arxiv.org/html/2608.07705#bib.bib27)\)\.

Finally, we outline practical mitigation strategies from both legal and technical perspectives, covering data sharing and model access, and highlight emerging legal proposals and potential regulatory adaptations\. By proposing a two\-dimensional, continuous, and context\-aware framework for assessing and addressing privacy risk, we aim to advance the scientific and clinical value of medical FMs while rigorously safeguarding patient privacy\. This review considers literature published up to March 2026, reflecting the state of the field at the time of writing\.

## 2Foundation Models in Healthcare

Healthcare foundation models can be categorized by the primary modality \(or modalities\) they are pretrained on \(e\.g\., clinical text, medical imaging\), and this matters for privacy because leakage from each modality can expose different types of sensitive information\. These modalities include, but are not limited to, structured electronic health records \(EHR\)\(Guoet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib9); Rencet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib12)\), clinical notes\(Xieet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib28); Yanget al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib29)\), medical images such as radiology scans\(Wuet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib30)\), and biological measures\(Thapaet al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib31); Liet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib32)\)\. A common FM development pathway aggregates data from multiple providers \(e\.g\., Hospital A/B/C\), followed by harmonization, de\-identification, and centralized or federated training to produce a foundation model intended for reuse across settings\(Jeonget al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib33)\)\. In many such pipelines, the pretraining data is de\-identified and HIPAA/GDPR compliant\(Bednarzet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib93)\)\.

Once trained, models are distributed in several ways \(Figure[1](https://arxiv.org/html/2608.07705#S2.F1)\)\. First, \(i\)Local deployment, where locally deployed copies are hosted within a health system’s secure environment\. In such settings, the hospital may use local identifiable data to fine\-tune the model, either as a one\-time process or continually, but the use is limited to the protected fire\-walled/certificate\-based system\(Rencet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib12); Junet al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib34)\)\. Alternatively, third\-party vendors may acquire clinical data to fine\-tune foundation models into downstream products, which are then deployed and integrated into hospital workflows\. In such settings, fine\-tuning can use identifiable or de\-identifiable\. In many such settings, deployment is limited to controlled releases within regulated environments, where the model is protected under safeguards similar to those applied to patient data\.

The second distribution type is \(ii\)Public release, in which the model is distributed as a web application with a user interface, exposed as a “black\-box” service via an application programming interface \(API\), or released as downloadable weights that can be used as\-is or further fine\-tuned\. These release channels differ materially in who can interrogate the model and under what controls, which in turn shapes the privacy threat surface\. For instance, interface\-based outputs may produce more heterogeneous outputs due to context loading, session history, and personalization effects, while APIs tend to be more controlled and reproducible\(Wanget al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib102)\)\. Furthermore, clinical language models are often continually pre\-trained or fine\-tuned from base models whose data mixture is unknown to developers\(Sellergrenet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib10)\)\. If these base models include sensitive health\-related data, they may unintentionally encode and leak identifiable patient information, even when downstream clinical training data is privacy\-compliant\.

Overall, differences in data modality, training setup, and model release strategy lead to different privacy risk profiles in healthcare foundation models\. Even when models are developed in compliance with HIPAA or GDPR, privacy risks can persist due to model capacity, fine\-tuning and model deployment\. To systematically analyse these privacy risks across deployment contexts, we introduce a two\-axis framework that characterizes how leakage arises and what information may be exposed\.

![Refer to caption](https://arxiv.org/html/2608.07705v1/x8.png)\(a\)Foundation model development and release workflow\.De\-identified patient data from multiple institutions \(e\.g\., hospitals\) is used to pre\-train a foundation model \(left\)\. For this model, two deployment pathways are shown: \(i\) Local deployment, where the foundation model is fine\-tuned by a healthcare institution or a third\-party business to be securely deployed within a hospital, or \(ii\) public release of the model as open weights or an interface for general users\. All stakeholders are assumed to operate within applicable legal frameworks, such as HIPAA, GDPR, and the EU AI Act\.
![Refer to caption](https://arxiv.org/html/2608.07705v1/x9.png)\(b\)Two\-dimensional privacy risk framework for model\-based data leakage\.The framework characterizes privacy risk of data leakage along two orthogonal axes\. Axis 1 \(x\-axis: Prior information\) captures what information is required to extract information from a model, ranging from no personal or public information \(e\.g\., demographics, unlinked social media traces\) to linkable personal information \(e\.g\. medications, diagnoses\)\. Axis 2 \(y\-axis: Leaked information\) captures what type of information can be extracted from the model, progressing from linkable personal information to identifiable personal information\.

Figure 1:Overview of the deployment workflow and privacy risk framework\.
## 3Contextualized Privacy Risk Assessment of Foundation Models

To better capture and understand the privacy risk of clinical foundation models, we characterize information leakage along two orthogonal axes \(Figure[1\(b\)](https://arxiv.org/html/2608.07705#S2.F1.sf2)\): \(i\) theprior informationthat initiates a leak, \(ii\) thetype and sensitivityof the information that is leaked by a model\. Together, these dimensions define a space of privacy risks that reflects both what the model exposes and the context in which that exposure is made possible\. In a practical analysis, privacy risk is determined jointly by what the model can leak and how likely it is that the required prior information is available or can be obtained in the relevant setting\.

### Axis I \- Prior information

The first axis captures the amount and type of prior information required for leakage to occur during inference\. This dimension is important for clinical risk assessment because the same technical vulnerability can imply very different real\-world harms\.

- •No prior information or publicly accessible information:At the lowest barrier, publicly obtainable information, such as public records, social media traces, or casual observation \(e\.g\., visible health status\) which may be non\-medical, would enable leakage\. A related scenario in this tier is unconditional generation, where the prompt does not start from a specific patient profile but instead probes the model to elicit memorized training content or population\-level information\. Unconditional generative models may memorize and reproduce training examples such as medical images, and carefully crafted prompts can sometimes extract snippets of real data or highly specific clinical patterns even from anonymized models\(Daret al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib61)\)\. This risk is especially salient as synthetic data cohorts are increasingly promoted as privacy\-preserving alternatives for data sharing\(Stadleret al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib62); Yoonet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib63); Sarkaret al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib64)\)\.
- •Linkable personal information:Leakage events in this tier require access to some information that is linked to an individual\. This category is practically important in real clinical environments, where a user may have access to a medication list, a single diagnostic test, or a clinical narrative from a referral note\. In such cases, the privacy risk arises from the model’s ability to infer additional sensitive details from these partial inputs, instead of information already held by the user\. Generative and multimodal foundation models can use sparse cues, such as a medical image or partial medication regimen, and elicit further details such as comorbidities, hospitalization patterns, or rare diagnoses\. The key point is that the input may be non\-identifying, while still enabling disclosure of sensitive information through model inference or synthesis\.

### Axis II \- Leaked information

Along the second axis, we categorize what type of information can be extracted from a medical foundation model\. This distinction matters because different leakage modes translate into different clinical and social harms\. Some disclosures are largely reputational or administrative, while others can expose diagnoses, treatments, or highly identifying clinical narratives specific to an individual\. Legal definitions of personal data distinguish between information whose processing is subject to data protection law and information that falls outside the scope of that law\.

Central to this distinction is whether an individual can be considered identifiable from the data, which determines how their processing is covered by the law\(Cobbe,[2026](https://arxiv.org/html/2608.07705#bib.bib54)\)\.

We group information leakage into two categories:

- •Linkable personal information:This category is similar to the definition in Axis I, but applied to a model leaking this information rather than an individual previously knowing it\. Even when foundation models are trained on de\-identified data, reconstruction of those de\-identified records is possible\(Packhäuseret al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib66)\)\. However, on its own, leakage of an isolated de\-identified piece of information poses a low risk, e\.g\., a generative model producing a replica of a patient’s de\-identified medical image\. The risk is introduced by linkability to other external information, where de\-identified data may be combined with \(quasi\-\)identifiers or external data to increase privacy risk\(Jeonget al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib33); Stadleret al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib62)\)\. In a model\-mediated leakage setting, reidentification risk with de\-identified data is amplified with additive sources\(Hallajet al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib75)\)\. For instance, in the generative model example, if it is also known that the model was trained on a cohort restricted to a specific condition \(e\.g\., a rare genetic disease registry\)\(Kehlet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib72)\), membership reveals diagnosis\.
- •Identifiable personal information:This includes any personal information that could render an individual identifiable, including data elements commonly removed during de\-identification \(name, address, etc\.\), which are discussed in greater detail in the next section on legal considerations\. For instance, generative models introduce an additional leakage pathway by directly synthesizing private training data50 and may synthesize memorized fragments of private training data, either verbatim or in near\-verbatim form\.

This two\-axis framework helps us locate data leakage in a practical, contextual risk space and is designed to focus on cases where the sensitivity of the leaked data is higher than the required prior information\. There are other possible marginal privacy loss scenarios\. For instance, in many membership\-inference settings, the user already holds a patient’s full \(or near\-full\) record and queries the model only to determine whether it was used in training\. In such cases, the key question is what additional information the model reveals beyond the prior knowledge, and which actors could plausibly obtain that starting point\. Other factors, such as model architecture, scale, and training procedures, further affect leakage possibility, shaping the risk that sensitive medical details could be reproduced or inferred in real\-world healthcare use\(Carlini and others,[2022](https://arxiv.org/html/2608.07705#bib.bib76)\)\.

Whether such leakage triggers legal obligations depends on whether the revealed information qualifies as personal data under applicable law\. We therefore examine how these leakage mechanisms intersect with existing legal frameworks in the United States and the European Union\.

## 4Legal Landscape

In the United States, the HIPAA Privacy Rule is the federal law that governs when certain forms of protected health information \(PHI\) may be shared\. For reasons we explain, it places only limited constraints relevant to the case studies we discuss\.

At the threshold, it applies only to “covered entities” and their “business associates\.” A covered entity is “\(1\) A health plan\[,\] \(2\) A health care clearinghouse \[, or\] \(3\) A health care provider who transmits any health information in electronic form in connection with a transaction covered by this subchapter\.” 45 C\.F\.R\. § 160\.103\. A business associate includes: “\(i\) A Health Information Organization, E–prescribing Gateway, or other person that provides data transmission services with respect to protected health information to a covered entity and that requires access on a routine basis to such protected health information\. \(ii\) A person that offers a personal health record to one or more individuals on behalf of a covered entity\. \(iii\) A subcontractor that creates, receives, maintains, or transmits protected health information on behalf of the business associate\.” 45 C\.F\.R\. § 160\.103\. This means that if the information is generated by someone other than a covered entity \(e\.g\., fitbit\) and does not leave the covered entity \(e\.g\., AI developed within a hospital system and deployed only within that hospital system\) or is shared with a business associate pursuant to the kind of business associate agreements recognized by HIPAA, there is no HIPAA problem\.

Second, HIPAA applies only to Protected Health Information \(PHI\), which is defined to mean “individually identifiable health information” \(IIHI\), which in turn is defined as: “information that is a subset of health information, including demographic information collected from an individual, and: \(1\) Is created or received by a health care provider, health plan, employer, or health care clearinghouse; and \(2\) Relates to the past, present, or future physical or mental health or condition of an individual; the provision of health care to an individual; or the past, present, or future payment for the provision of health care to an individual; and \(i\) That identifies the individual; or \(ii\) With respect to which there is a reasonable basis to believe the information can be used to identify the individual\.” 45 C\.F\.R\. § 160\.103 If the information shared to train a model does not meet this definition, then HIPAA does not prevent a covered entity from sharing it to train a model\.

Third, HIPAA specifies that “\[h\]ealth information that does not identify an individual and with respect to which there is no reasonable basis to believe that the information can be used to identify an individual is not individually identifiable health information\.” 45 C\.F\.R\. § 164\.514\. More specifically, it provides two methods by which data can meet that requirement\. The first is the so\-called “expert determination” pathway, in which a “person with appropriate knowledge of and experience with generally accepted statistical and scientific principles and methods for rendering information not individually identifiable\.\(i\) Applying such principles and methods, determines that the risk is very small that the information could be used, alone or in combination with other reasonably available information, by an anticipated recipient to identify an individual who is a subject of the information; and \(ii\) Documents the methods and results of the analysis that justify such determination\.” 45 C\.F\.R\. § 164\.514\. This method is not commonly used, but the examples we discuss in this paper raise the question of under what circumstances an expert could make the necessary determination, given the increased data leakage risks\. Second, and more commonly used, HIPAA specifies that information ceases to be individually identifiable health information if 18 categories of identifiers are stripped before sharing\. 45 C\.F\.R\. § 164\.514\. In the EU, the GDPR applies to any processing of personal data, including model training and deployment24, where personal data means “any information relating to an identified or identifiable person” \(Art\. 4\(1\)\)\. Whether de\-identified data qualifies as “identifiable” under the GDPR depends on the risk of re\-identification\. Recent jurisprudence from the European Court of Justice \(see Appendix[Appendix A](https://arxiv.org/html/2608.07705#Sx2)\) clarifies that this assessment must be made from the perspective of the relevant controller or recipient\. The analysis considers the means reasonably available to that entity to re\-identify the data\. Truly anonymized data falls outside the GDPR, whereas pseudonymized data generally remains within scope\. Personal health data constitute a special category of personal data whose processing requires a lawful basis and specific justification, and EU member states may impose additional processing restrictions \(Art\. 9\)\. Where personal data may appear in training data, prompts, and logs, adherence to data\-protection principles such as purpose limitation, data minimization, transparency, and security must be ensured\(Bodnari and Travis,[2025](https://arxiv.org/html/2608.07705#bib.bib101)\)\. The GDPR also describes roles that determine the legal responsibilities of various parties \(TableLABEL:tab:privacy\_scenarios\)\. The naming depends on the actual influence over purpose and means, not contractual labels\. This means that a vendor called a “processor” may become a “joint controller” if it meaningfully influences model architecture or secondary use\.

Table 1:Description of GDPR roles for responsibility in protecting patient privacy\.The EU AI Act \(AIA\) introduces tiered obligations based on application risk at the level of AI models and AI systems, including for general\-purpose AI \(GPAI\)53\. AI systems that require a third\-party conformity assessment according to the Medical Device Regulation \(MDR\), i\.e\., systems such as medical AI software assigned class IIa or higher under the MDR, are automatically high\-risk systems under the AIA \(Annex I Section A No\. 11 AIA\)\. Other elements of the EU regulatory ecosystem are listed in Appendix[Appendix B](https://arxiv.org/html/2608.07705#Sx3)\. Beyond statutory privacy regimes in the US and EU, model development and deployment are shaped by a broader governance ecosystem that includes contracts, institutional oversight, technical standards \(e\.g\., ISO/IEC\), case law, and regulatory guidance\. While formal law carries binding force but evolves slowly, guidance and standards can adapt more quickly, though they may differ across jurisdictions and lack uniform enforcement\.

Data and model governancealso rely on contracts and institutional controls that vary across the development pipeline\(Morleyet al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib43); Daviset al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib44)\)\. Where foundation model development qualifies as human subject research under applicable regulations, activities may be within the scope of institutional review board \(IRB\) oversight and data use agreement\(Spector\-Bagdadyet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib58)\)\. However, regulatory requirements differ across jurisdictions\. In many EU member states, secondary use of health data does not necessarily require approval from a research ethics committee \(REC\), and the applicable GDPR legal basis for data preprocessing remains inconsistently interpreted across regulators and other institutions\(Smitet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib48); European Commission,[2021](https://arxiv.org/html/2608.07705#bib.bib49)\)\. However, many foundation model projects, particularly those relying on retrospectively collected or de\-identified hospital datasets shared with third\-party developers, are structured as secondary data use or product development activities and may fall outside formal IRB review\(Allenet al\.,[2014](https://arxiv.org/html/2608.07705#bib.bib100)\)\. In such cases, governance is primarily contractually agreed on rather than regulatory\. This landscape may evolve with the upcoming European Health Data Space \(EHDS\) regulation, which is expected to introduce permit\-based access mechanisms for secondary use of health data, although implementation details are still emerging\(European Data Protection Board and European Data Protection Supervisor,[2026](https://arxiv.org/html/2608.07705#bib.bib99); Becker and Dove,[2026](https://arxiv.org/html/2608.07705#bib.bib47)\)\. Governance requirements also vary depending on whether models are vendor\-built, co\-developed, or developed internally62\. At model release, oversight often shifts toward licensing terms, which may restrict re\-identification or sensitive inference\(Jungkunzet al\.,[2021](https://arxiv.org/html/2608.07705#bib.bib59)\)\.

## 5Risk analysis case studies

Decision makers can use the two\-axis framework to compare privacy risk across realistic deployment and use scenarios by jointly considering how sensitive the potential leakage is and how plausible it is that the necessary prior information would be available or obtainable in that context\. This operational view supports scenario\-specific governance: the same model behavior may imply very different risks depending on who can query the model, what auxiliary data are accessible, and what controls constrain use\. With this framing, in this section, we present various leakage scenarios and analyze their risks along our two dimensions\. These scenarios are summarized in TableLABEL:tab:privacy\_scenariosand Figure[2](https://arxiv.org/html/2608.07705#S5.F2), and are positioned within our two\-axis risk space in Figure[1\(b\)](https://arxiv.org/html/2608.07705#S2.F1.sf2)\.

![Refer to caption](https://arxiv.org/html/2608.07705v1/x10.png)Figure 2:Privacy leakage scenarios across deployment pathways\. Illustrative examples of the six scenarios described in TableLABEL:tab:privacy\_scenarios, spanning common deployment pathways for healthcare foundation models\. Panels are grouped by deployment setting \(pink: public release; purple: local/hospital deployment\) and highlight distinct release/deployment routes within each setting, including releasing an API \(S1\), releasing a web interface \(S2\), deploying a third\-party model within a hospital environment \(S3\), using a locally fine\-tuned hospital model \(S4\), and public release of model weights \(S5\)\. Data originally consented for clinical research are later reused to train a general\-purpose AI system without renewed consent \(S6\)\.Each row in Figure[2](https://arxiv.org/html/2608.07705#S5.F2)presents an example scenario of data leakage that belongs to the different deployment pathway depicted in Figure[1](https://arxiv.org/html/2608.07705#S2.F1), and is color\-coded to show what use\-case the example belongs to, i\.e\., controlled clinical deployment \(purple\) or public release \(pink\)\. The “intended use” defines the purpose, which shapes regulatory obligations\. The “training data” clarifies what kinds of data were used, affecting whether personal or special\-category data are involved\. The “privacy risk analysis” details the risk associated with data leakage as a function of the prior information\. Finally, “GDPR/AIA Risk Analysis” identifies the potential lawful grounds in GDPR \(roles explained in Table[1](https://arxiv.org/html/2608.07705#S4.T1)\) for each specific scenario\.

For the case studies in this paper, with one caveat, so long as the covered entity strips all these identifiers before sharing, HIPAA would not be violated in using this data to train a model\. The caveat is that HIPAA treats the stripping of identifiers as sufficient if and only if “\[t\]he covered entity does not have actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information\.” 45 C\.F\.R\. § 164\.514\. The meaning of this language has not received much elaboration by the U\.S\. Department of Health and Human Services \(HHS\) nor by scholars\. The kind of data leakage we discuss in this paper may provide covered entities “actual knowledge” about the way “in combination with other information, could be used “to identify an individual,” but this is uncertain\. There is also a provision allowing the stripping of some but not all identifiers to form a “limited data set \[…\] if the covered entity enters into a data use agreement with the limited data set recipient in the way set out by HIPAA\.

Table 2:Example scenarios of data leakage across various deployment pathways\.Example scenariosS1 \- Reconstruction:A company releases a brain\-MRI generator model \(as an API or open weights\), that takes a short prompt like: “Generate an axial T1 brain MRI with mild motion artifact\.” Training data included de\-identified MRIs from EU hospitals or EU patients\. During routine use, a user requests a handful of images with similarly generic prompts \(no patient name, age, or identifiers\)\. One generated image is a near\-replica of a real scan from a training individual\. The replica preserves distinctive anatomy and pathological patterns that could enable re\-identification\.Intended use:Image generation for research or educational purposes, not to reproduce real patient scans\.Training data:De\-identified brain\-MRI images\.Privacy risk analysis:A single leaked MRI contains limited information in isolation\. Risk increases if the image can be linked to auxiliary information about the training cohort \(e\.g\., that the model was trained on MRIs from Hospital X’s tumor clinic\), and especially if membership inference confirms that a specific person’s scan was included\.GDPR/AIA Risk Analysis:The entity that trains and releases the model acts as thecontrollerfor the training data and must ensure lawful processing and effective safeguards against memorization\. If a separate provider deploys the model via API, it may qualify as an additionalcontrollerfor outputs\. Downstream users are typically recipients unless they further process or redistribute data\.GDPRArt\. 5\(1\)\(f\) – integrity and confidentialityArt\. 25 – data protection by designArt\. 32 – security safeguardsArt\. 9 MRI Health data → special categoryEven “de\-identified” MRIs can be personal data if linkage is reasonably possible \(Recital 26\) \(as discussed in Section 5\)\.AI Act:GPAI / generative models must document training sources and mitigate memorization risk, but the Act does not define when outputs become personal data\.S2 \- Memorization:A company offers a public user interface for clinical\-note generation\. A user provides a prompt like: “Write an example triage note for a woman in her 30s seen at Hospital X in March 2024 with findings X/Y/Z\.” The patient has posted online clues \(a sleep\-lab photo, hospital location, distinctive symptoms\)\. The model returns text closely matching a real training note, releasing sensitive information the person from the training cohort never shared publicly\.Intended use:Clinical note drafting support, not reproducing real patient histories\.Training data:De\-identified triage chats and clinical notes\.Privacy risk analysis:A single generated note may not contain direct identifiers, yet risk arises when outputs can be linked to external context\. For example, combining model output with publicly available clues \(such as a patient posting about participation in a sleep study\) may enable attribution and reveal additional sensitive information, including training\-set membership\. Near\-verbatim reproduction is consistent with documented evidence that LLMs can memorize and reproduce training data under certain prompts\.GDPR/AIA Risk Analysis:The company training and offering the API acts ascontrollerfor training and deployment and must ensure that de\-identification and safeguards are effective\. Contractual or research\-use permissions do not override GDPR obligations if outputs are personal data\.GDPRArt\. 5\(1\)\(f\) – integrity and confidentialityArt\. 25 – data protection by designArt\. 32 – security safeguardsArt\. 9 Health data → special categoryThe legal implications depend on whether the output becomes identifiable in context \(see Section 5\)\.S3 \- Leaked autocompletion:A hospital purchases an EHR “smart note” model from a third\-party company that clinicians can use the LLM to draft a patient summary\. The model was fine\-tuned on past discharge summaries that still contained identifiers\. While writing Patient A’s discharge note, the clinician types: “Follow up with cardiology, Dr\. …” and accepts an autocomplete suggestion\. The model inserts a line copied from a different training example: “Follow up with Dr\. X next Tuesday — call 416\-XXX\-XXXX\. Patient: John D\.” which belongs to Patient B\. That PHI is now mistakenly placed into Patient A’s chart\.Intended use:The LLM supports discharge summary drafting for patients\.Training data:Trained on de\-identified patient notes and finetuned on identifiable discharge summaries from the hospital\.Privacy risk analysis:The incident covers a disclosure of PHI as the model reproduces identifiers from training data\. This leakage stems from model behaviors under normal use, rather than unauthorized access\.GDPR/AIA Risk Analysis:The hospital, as the datacontrollerand model deployer, remains responsible for preventing such disclosures through appropriate safeguards, regardless of whether fine\-tuning was internal or outsourced\. Vendors involved in model training or hosting shareprocessororjoint controllerobligations\.GDPRArt\. 5\(1\)\(f\) – integrity and confidentialityArt\. 9 – special\-category data disclosureArt\. 25 – insufficient privacy\-by\-designArt\. 32 – inadequate safeguardsArt\. 33/34 – potential breach notificationAI ActHigh\-risk clinical systems require risk management and human oversight but do not define reconstruction thresholds\.S4 \- Autocompletion:A hospital deploys an EHR foundation model as a documentation assistant within a patient chart\. The system receives structured context \(demographics, recent lab results, medication list and prior notes\) and autogenerates a draft summary for clinical handoff\.draft the likely recent timeline to support handoff”\)\. A rare case \(female with markedly elevated D\-dimer and atypical coagulation markers\) the model generates a highly specific narrative including an uncommon diagnostic pathway and medication regimen\. Although framed as a probabilistic completion, the rare combination of features and highly specific management steps may mirror a distinctive historical case from the training data\.Intended use:Clinical tool for context completion, not retrieval of real patient histories\.Training data:Trained on de\-identified patient notes and finetuned on identifiable discharge summaries from the hospital\.Privacy risk analysis:If the completion stays at the level of common clinical reasoning \(e\.g\., “consider PE workup”\), privacy risk is low\. However, rare or low\-frequency combinations of diagnosis steps, or medication dosing patterns can render the output record\-like and plausibly attributable to a specific individual\. At that point, the risk shifts from a generic inference to plausible reconstruction\.GDPR/AIA Risk Analysis:Where the hospital determines the purpose and means of deployment, it acts ascontroller\. The developer may qualify asprocessororjoint controller, depending on its influence over training and system design\. Both must ensure safeguards against reconstruction and memorization\.GDPRArt\. 5\(1\)\(f\) – integrity and confidentialityArt\. 25 – data protection by designArt\. 32 – security safeguardsArt\. 9 Health data → special categoryArt\. 33–34 Could trigger breach notification\.Art\. 4\(2\) Reconstruction = processing/disclosure of personal dataAI ActHigh\-risk clinical systems require risk management and human oversight but do not define reconstruction thresholds\.Table 2:Example scenarios of data leakage across various deployment pathways \(continued\)\.Example scenariosS5 \- Membership leakage:To assess how training data influences model behavior, a researcher probes a publicly released histopathology foundation model trained on a breast cancer cohort\. Using a de\-identified digital slide patch from a known patient, they observe that the model’s embedding similarity scores and calibration behavior are unusually consistent with “seen during training” examples compared to matched controls\. Even without revealing the patient’s label or reproducing any identifiable image content, the interaction can support inference that this individual’s specimen was part of the training cohort\.Intended use:Clinical language support, not confirmation of dataset\.Training data:Trained on de\-identified patient data\.Privacy risk analysis:A membership signal does not reveal full records but indicates that a person’s data was used for model training\. In health care, inclusion alone may reveal sensitive facts, especially where the cohort is condition\-specific \(e\.g\., cancer patients\)\. Membership can therefore amount to disclosure of special\-category data\.GDPR/AIA Risk Analysis:The model provider acts ascontrollerfor training and deployment decisions and must mitigate such risks\. API deployers share responsibility if their system exposes signals that enable inference\. Users performing tests are typically recipients unless they systematically collect or publish results\.GDPRArt\. 5\(1\)\(f\) – integrity and confidentialityArt\. 25 – data protection by designArt\. 32 – security safeguardsArt\. 9 Health data → special categoryAs noted above, identifiability under GDPR is assessed relative to the actor’s capabilities and accessible auxiliary information\.AI ActThe AI Act requires risk mitigation for sensitive\-data memorization but does not address membership inference explicitly\.S6 \- Secondary use:Patients at Hospital X consent to their clinical data being used for medical research\. Years later, the hospital \(or a partner institution\) contributed a large corpus of historical EHR notes and imaging data to a foundation\-model pretraining dataset\. The resulting model is used for broad clinical and commercial applications, including deployment by third parties\. Patients were never explicitly informed that their data might be used to train general\-purpose AI systems or commercial foundation models\. Some patients later withdraw consent, but their data have already influenced the model’s weights\.Intended use:Original: medical researchLater: general\-purpose model development and downstream deployment\.Training data:De\-identified patient data\.Privacy risk analysis:This scenario covers a purpose drift rather than a direct disclosure: data are reused beyond what patients reasonably expected\. Once embedded in model weights, data are difficult to remove, creating tension with erasure rights\.GDPR/AIA Risk Analysis:The hospital, ascontrollerand must ensure lawful secondary use and compatibility with the original purpose\. Developers may qualify as controllers or joint controllers if they define training objectives\. Responsibility centers on ensuring valid legal bases, transparency, and safeguards before reuse\.GDPRArt\. 5\(1\)\(b\) – purpose limitationArt\. 17 – right to erasureArt\. 9 Health data → special categoryArt\. 6\(4\) permits secondary research use with safeguards and requires compatibility assessmentConsent limits: Recital 33 allows broad consent but not unlimited future use\.Transparency obligations \(Art\. 13–14\)\.AI ActThe AI Act requires data governance and documentation but does not resolve issues related to consent scope or machine unlearning\.Note, the scenarios in TableLABEL:tab:privacy\_scenariosdescribe AI systems used to provide information for clinical decision\-making, such as diagnosis, risk stratification, or treatment planning\. When intended for such purposes, they qualify as medical device software under Rule 11 of Annex VIII to the EU Medical Device Regulation and are consequently classified as high\-risk AI systems under Annex I Section A of the EU AI Act \(more details see Section[4](https://arxiv.org/html/2608.07705#S4)\)\. Across frameworks, privacy law does not define quantitative thresholds for memorization, reconstruction, or membership inference\. This technology\-neutral design is understandable, but it creates uncertainty when evaluating whether a model remains effectively anonymized or legally compliant\. Throughout the scenarios, this gap recurs: legal obligations attach to disclosure of personal data, yet models may reveal probabilistic or fragmentary signals that fall short of explicit record release\.

Taken together, these scenarios illustrate how privacy risks can arise across different stages of the foundation model lifecycle and deployment settings\. Next, we therefore outline technical and legal mitigation strategies that can reduce these risks at different points in the pipeline\.

## 6Mitigating Privacy Risks

Privacy risks in medical foundation models arise across the full lifecycle, from data collection and curation to model development\. They therefore cannot be addressed with a single safeguard at one point in the pipeline\. Effective mitigation requires coordinated legal and technical measures throughout the lifecycle\. The two\-axis framework also helps clarify how to prioritize these measures\. Scenarios involving more sensitive leakage and requiring little prior information demand stronger safeguards\. Mitigation should thus be calibrated both to the lifecycle stage and to a contextual risk space\. Below, we discuss these mitigation strategies at the various levels of model development\.

### Data collection

Legal frameworks such as the GDPR and HIPAA emphasize some combination of lawful basis, purpose limitation, data minimization, and de\-identification or pseudonymization as primary safeguards\(Morleyet al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib43); Yanget al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib84)\)\. These requirements are largely formulated around the release and sharing of datasets, rather than downstream model behavior\. As mentioned earlier, HIPAA applies only to PHI that identifies the individual, or with respect to which there is a reasonable basis to believe the information can be used to identify the individual\. If the information shared to train a model does not meet this definition, then HIPAA does not prevent a covered entity from sharing it to train a model\. HIPAA also puts in place a general “minimum necessary” constraint, that a covered entity “must make reasonable efforts to limit protected health information to the minimum necessary to accomplish the intended purpose of the use, disclosure, or request\.” 45 C\.F\.R\. § 164\.502\. This requirement would govern any sharing of data by a covered entity\. There is also a provision allowing the stripping of some but not all identifiers to form a “limited data set \[…\] if the covered entity enters into a data use agreement with the limited data set recipient” in the way set out by HIPAA\.

Recent regulatory and policy developments reflect growing interest in enabling secondary data use while managing privacy risk, including proposals for decentralized architectures such as federated personal health data spaces that keep individuals “in the loop” while still supporting large\-scale health data use\(Raabet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib85)\)\. Parallel initiatives, such as the AI Act’s risk\-based classification of medical AI, the EHDS and Data Act frameworks, and the Commission’s Digital Omnibus proposal, further signal a shift toward facilitating data sharing and AI training through permits, opt\-outs, and contractual mechanisms\. Recently, the French DPA \(CNIL\) shared a decision aid regarding when a model should be considered anonymous, meaning outside the scope of the GDPR\(Commission Nationale de l’Informatique et des Libertés \(CNIL\),[2026](https://arxiv.org/html/2608.07705#bib.bib77)\)\. Taken together, these developments suggest a broader move away from strict consent\-based models and absolute notions of personal data toward tiered, context\-dependent privacy protections, motivating dynamic risk assessments aligned with practical leakage scenarios\.

Practical steps at the data curation stage can reduce downstream leakage risk\. Variability in de\-identification practices, underlying data distributions, and dataset sizes may additionally introduce privacy risks\. For example, if one site de\-identifies through reversible pseudonymization, such as “hidden in plain sight” while another site uses anonymization through masking, these differences may result in uneven privacy guarantees across contributors and enable source inference attacks\(Huet al\.,[2021](https://arxiv.org/html/2608.07705#bib.bib79)\)\. Increasing dataset size and diversity is an effective mitigation and reduces the likelihood that rare samples dominate leakage positions high on axis II in Figure[1\(b\)](https://arxiv.org/html/2608.07705#S2.F1.sf2)\. Generative models exhibit stronger memorization when trained on smaller or homogeneous datasets\(Guet al\.,[2023](https://arxiv.org/html/2608.07705#bib.bib80)\), whereas larger and diverse datasets promote generalization\. Memorization and extractability are often concentrated on rare samples, where low\-frequency patterns are more likely to be reproduced under targeted prompting\(Kulynychet al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib51)\)\. Constructing large, heterogeneous, multi\-institutional cohorts can therefore dilute exposure to any individual record and reduce the feasibility of linkage\-based identification in practice\.

### Model development

Modern privacy frameworks and guidance emphasize risk management and data protection by design, but offer limited guidance on privacy harms arising from model behavior rather than explicit data disclosure\. This creates a compliance gap: standards such as ISO/IEC 42001:2023 require “appropriate” safeguards, yet do not specify what constitutes actionable leakage, what evidence is sufficient to demonstrate safety, or when remediation is mandatory\. In the U\.S\., outside HIPAA\-covered entities and business associates, there is often no general restriction on using health\-related data for model training\. Still, privacy\-violating outputs may expose developers to common\-law privacy claims, an area that remains underdeveloped\.

In practice, these obligations are operationalized through privacy\-specific evaluations and defenses\. Leakage auditing, including membership inference, extraction tests, canary exposure, and structured red\-teaming, has become a common practice\(Tonekaboniet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib65); Perezet al\.,[2022](https://arxiv.org/html/2608.07705#bib.bib45); Meeuset al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib46); Purpuraet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib50)\)\. Developers also deploy alignment and safety layers\(Liet al\.,[2024](https://arxiv.org/html/2608.07705#bib.bib81)\)to filter sensitive generations before release\. Differential privacy \(DP\)\(Dwork and Roth,[2014](https://arxiv.org/html/2608.07705#bib.bib82)\)provides formal bounds on information leakage and has been shown to reduce membership inference and reconstruction risks in health and deep learning settings\(Abadiet al\.,[2016](https://arxiv.org/html/2608.07705#bib.bib83); McMahanet al\.,[2018](https://arxiv.org/html/2608.07705#bib.bib86)\)\. DP primarily reduces vertical movement on axis II in Figure[1\(b\)](https://arxiv.org/html/2608.07705#S2.F1.sf2)and limits how much new personal information the model can reveal, regardless of prior knowledge\. Other approaches, such as hierarchical memories\(Pouransariet al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib87)\), or isolating memorization by design\(Ghosalet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib88)\), further illustrate available technical controls\.

Remediation remains under\-specified\. Although safety fine\-tuning and targeted retraining are feasible, legal frameworks typically do not mandate ongoing post\-training evaluation to verify that memorized content has been removed, nor do they define thresholds that trigger corrective action short of full record reconstruction\. The result is a persistent gap between detecting leakage and enforcing mitigation, highlighting the need for concrete, testable criteria that link high\-level legal duties to measurable model behavior\.

### Model deployment or release

At deployment, legal frameworks typically require appropriate technical and organizational safeguards to protect patient privacy, often implemented through licenses, terms of use, and access controls\. In practice, however, privacy risk depends heavily on deployment format\. Safeguards may include graduated access controls, rate limiting, logging, architectural separation between high\-capability models and public interfaces, and monitoring for anomalous or extraction\-like use\. By contrast, unrestricted APIs, downloadable weights, or open fine\-tuning interfaces enable model extraction, repurposing, and redistribution\(Papernotet al\.,[2016](https://arxiv.org/html/2608.07705#bib.bib89)\)beyond the original governance context, potentially bypassing training\-time safeguards, such as output filtering or privacy\-preserving optimization\.

U\.S\. tort law, which provides the ability to bring a claim against a wrongdoer after an injury occurs, is still developing in this area and it is not clear if it is likely to yield robust liability\(De Bruyne and Van Leenhove,[2022](https://arxiv.org/html/2608.07705#bib.bib104); Noto La Diega and Bezerra,[2024](https://arxiv.org/html/2608.07705#bib.bib105)\)\. Importantly, HIPAA does not provide patients a private right of action\. That means patients may not themselves sue directly under the statute if their information is improperly shared – only HHS may bring such an action\. However, several state courts have recognized state common law torts in instances where there is “unauthorized disclosure of confidential information obtained in the course of \[a medical treatment\] relationship,” and have relied on HIPAA as informing the relevant standard of care\. Byrne v\. Avery Ctr\. for Obstetrics & Gynecology, P\.C\., 327 Conn\. 540 \(2018\)\. Private individuals may, in some instances, be able to bring such cases, though this case law is emerging, and the application to cases involving data leakage of the kind we discuss would be novel\. When, due to leakage, identifiable information about a patient is revealed they may also be able to bring one of the so\-called “privacy torts,” most likely an action for “public disclosure of private facts” involving “\(1\) the public disclosure, \(2\) of a private fact, \(3\) that would be offensive and objectionable to a reasonable person, and \(4\) that is not of legitimate public concern\.” 103 Am\. Jur\. Proof of Facts 3d 159\. We are not aware of any instance where this tort has been advanced in a case where generative AI produces true private facts about an individual due to leakage, so the theory is untested\. Among the questions that courts might face is whether unintentional sharing counts for the tort or intentional action is required, and what kinds of injuries are cognizable, and the answers might vary by state\. Even if plaintiffs are successful in suing model developers or implementers in such cases, common law judge\-by\-judge decision\-making in these areas is unlikely to generate detailed guidance in the short term\. Finally, beyond HIPAA, some states have additional privacy statutes that may further limit the sharing of data\.

Technical mitigation continues after release\. Post\-deployment auditing, including canary insertion, targeted extraction probes, or memorization benchmarks, can detect reproduction of rare or sensitive records\(Pandaet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib90)\)\. Maintenance mechanisms such as machine unlearning\(Liet al\.,[2026](https://arxiv.org/html/2608.07705#bib.bib91)\)and periodic retraining offer a way to reduce memorization in response to new risks or regulatory requests\. Policy discussions increasingly emphasize such layered monitoring\(International AI Safety Report,[2026](https://arxiv.org/html/2608.07705#bib.bib92)\)\. Recent EU reform proposals, including amendments to Article 9 GDPR in the Digital Omnibus package, if adopted, may introduce additional obligations for controllers\. Where special\-category personal data are identified in training datasets or models, controllers must remove them or, if removal would require disproportionate effort, effectively prevent their use in outputs or being disclosed85\. While it remains unclear whether and in what form this approach will become binding law, it reflects a shift toward output\-level containment where full deletion is technically infeasible\.

## 7Conclusion

Building on the deployment pathways, risk framework, and scenario analysis presented above, medical foundation models offer substantial potential for clinical decision support, screening, and public health, but their safe deployment requires privacy evaluation that goes beyond controlling inputs and access\. Privacy risk in these systems is continuous rather than categorical\. In this work, we introduce a structured, two\-dimensional framework for assessing model\-mediated privacy risk by jointly characterizing \(i\) the prior\-knowledge threshold needed to elicit leakage and \(ii\) the type and extent of information that can be revealed\. This framing provides a practical lens for comparing risk across deployment settings \(e\.g\., controlled clinical use versus broader release\), identifying plausible leakage pathways, and mapping them to targeted mitigation strategies and governance requirements\. It helps align technical risk assessment with emerging regulatory expectations, particularly as health\-related AI systems expand beyond traditional clinical settings\. Although our focus is privacy leakage, these risks intersect with broader safety concerns such as bias propagation, misinformation, and downstream misuse, highlighting the need for lifecycle\-aware evaluation rather than isolated technical safeguards\.

At present, EU privacy law largely differentiates cases based on whether an individual is identifiable\. Our framework suggests that privacy risk is more nuanced: harm may arise not only from clear identification, but also from the sensitivity of leaked information, the ease with which inferences can be drawn, or reconstruction\-style outputs that resemble real records without directly naming a person\. This raises a broader question for future work: whether privacy law should continue to rely so heavily on identifiability alone, or whether legal standards could evolve to better reflect gradations of inference, sensitivity, and probabilistic disclosure captured by a structured risk framework\. The legal and regulatory landscape for AI remains in flux\. A notable trend in the EU is the facilitation of AI training through expanded data\-access and secondary\-use provisions\. While these developments may accelerate innovation, they also raise concerns about the erosion of privacy safeguards if protections do not evolve in parallel\. How these regulatory trajectories unfold will play a key role in striking a balance between innovation, patient safety, and data protection\.

Recent developments in U\.S\. health\-data governance illustrate both expanding privacy protections and efforts to facilitate public\-health data sharing\. In particular, several states are introducing privacy rules that extend beyond the traditional scope of HIPAA\. For example, California has strengthened protections for certain health\-related and location data, such as restrictions on geofencing around health\-care facilities, and expanded privacy obligations to entities outside HIPAA’s coverage\(California Department of Information Integrity,[2024](https://arxiv.org/html/2608.07705#bib.bib52); Alder,[2025](https://arxiv.org/html/2608.07705#bib.bib53)\)\. At the same time, state privacy laws such as the CCPA generally exempt protected health information regulated under HIPAA, while still applying to other categories of personal data that fall outside HIPAA’s definition88\.

Finally, although our focus is individual\-level privacy, model\-mediated leakage can also generate collective harms, including profiling and differential treatment of groups\(Bednarzet al\.,[2025](https://arxiv.org/html/2608.07705#bib.bib93)\)\. Future work should examine how such risks propagate beyond individuals to affect populations and social systems\. It should also analyze how legal roles under frameworks such as the GDPR and AI Act may shift in more complex settings, for example, in federated learning or fine\-tuning arrangements\.

## Acknowledgment

#### Funding:

The Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, Postdoctoral fellowship \(ST\)\. Wallenberg Foundation Scholarship Program for Postdoctoral studies at Massachusetts Institute of Technology and the Broad Institute \(LS\)\. Novo Nordisk Foundation Grant for a scientifically independent International Collaborative Bioscience Innovation & Law Programme \(Inter\-CeBIL Programme, grant number NNF23SA0087056\) \(IGC\)\. The Weill Cancer Hub West Initiative \(SR, EA\)\. ELLIS Institute Finland \(CC\)\. National Science Foundation \(NSF\) 22\-586 Faculty Early Career Development Award \(no\.2339381\) \(MG\)\. The AI2050 Program at Schmidt Sciences \(MG\)\.

#### Author’s contributions:

Conceptualization: ST, LS, MG Supervision: MG, CC, EA, IGC Methodology: ST, LS, SR, CC, IGC, EA, MG Visualization: ST, LS Writing \- original draft: ST, LS Writing \- review & editing: ST, LS, SR, CC, IGC, EA, MG

#### Competing interests:

IGC is a member of the Bayer Bioethics Council, a bioethics advisor for Bexorg, and an advisor for World Class Health and Manhattan Neuroscience LLP\. He recently concluded service as the chair of the ethics advisory board for Illumina\. He was also compensated for speaking at events organized by Philips with the Washington Post as well as the Doctors Company, attending the Transformational Therapeutics Leadership Forum organized by Galen Atlantica, and retained as an expert in health privacy, gender\-affirming care, and reproductive technology lawsuits\.

#### Data, code, and materials availability:

No custom code, datasets, or unique materials were generated for this study\. All information supporting the conclusions is contained in the manuscript\.

## References

- M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. Zhang \(2016\)Deep learning with differential privacy\.InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security,pp\. 308–318\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- S\. Alder \(2025\)What is the ccpa hipaa exemption?\.Note:[https://www\.hipaajournal\.com/ccpa\-hipaa\-exemption/](https://www.hipaajournal.com/ccpa-hipaa-exemption/)Accessed March 17, 2026Cited by:[§7](https://arxiv.org/html/2608.07705#S7.p3.1)\.
- C\. G\. Allen, T\. R\. Des Jardins, A\. Heider, K\. A\. Lyman, J\. McWilliams, A\. L\. Rein, A\. A\. Schachter, S\. Silow\-Carroll, A\. Wright, and C\. P\. Friedman \(2014\)Data governance and data sharing agreements for community\-wide health information exchange: lessons from the beacon communities\.eGEMs2\(1\),pp\. 1057\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- I\. Barberá \(2025\)AI privacy risks & mitigations – large language models \(LLMs\)\.Note:European Data Protection Board Support Pool of ExpertsAccessed 26 February 2026External Links:[Link](https://www.edpb.europa.eu/system/files/2025-04/ai-privacy-risks-and-mitigations-in-llms.pdf)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1),[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- R\. Becker and E\. S\. Dove \(2026\)The eu gdpr and secondary use of health and genetic data for research support purposes\.International Data Privacy Law16\(2\),pp\. ipag001\.External Links:[Document](https://dx.doi.org/10.1093/idpl/ipag001),[Link](https://doi.org/10.1093/idpl/ipag001)Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- Z\. Bednarz, K\. Lewis, and J\. Sadowski \(2025\)‘It’s not personal, it’s strictly business’: behavioural insurance and the impacts of non\-personal data on individuals, groups and societies\.Computer Law & Security Review56,pp\. 106096\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1),[§7](https://arxiv.org/html/2608.07705#S7.p4.1)\.
- K\. Benitez and B\. Malin \(2010\)Evaluating re\-identification risks with respect to the HIPAA privacy rule\.Journal of the American Medical Informatics Association17\(2\),pp\. 169–177\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1),[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- A\. Bhargava, C\. López\-Espina, L\. Schmalz, S\. Khan, G\. L\. Watson, D\. Urdiales, L\. Updike, N\. Kurtzman, A\. Dagan, A\. Doodlesack, A\. Espinosa, A\. Halalau, C\. DeMarco, F\. Davila, H\. Davila, M\. Sims, N\. Maddens, R\. Berghea, and S\. Smith \(2024\)FDA\-authorized ai/ml tool for sepsis prediction: development and validation\.NEJM AI\.External Links:[Document](https://dx.doi.org/10.1056/AIoa2400867)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- A\. Bodnari and J\. Travis \(2025\)Scaling enterprise ai in healthcare: the role of governance in risk mitigation frameworks\.npj Digital Medicine8\(1\),pp\. 272\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p4.1)\.
- California Department of Information Integrity \(2024\)Federal and state health laws\.Note:[https://www\.cdii\.ca\.gov/compliance\-and\-policy/resources/federal\-and\-state\-health\-laws/](https://www.cdii.ca.gov/compliance-and-policy/resources/federal-and-state-health-laws/)Accessed March 16, 2026Cited by:[§7](https://arxiv.org/html/2608.07705#S7.p3.1)\.
- N\. Carlini, C\. Liu, Ú\. Erlingsson, J\. Kos, and D\. Song \(2019\)The secret sharer: evaluating and testing unintended memorization in neural networks\.In28th USENIX Security Symposium \(USENIX Security 19\),pp\. 267–284\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- N\. Carliniet al\.\(2022\)Quantifying memorization across neural language models\.arXiv preprint arXiv:2202\.07646\.Cited by:[§3](https://arxiv.org/html/2608.07705#S3.SSx2.p4.1)\.
- J\. Cobbe \(2026\)A relative mess: identifying data subjects in multi\-party processing\.Computer Law & Security Review62,pp\. 106362\.Cited by:[§3](https://arxiv.org/html/2608.07705#S3.SSx2.p2.1)\.
- Commission Nationale de l’Informatique et des Libertés \(CNIL\) \(2026\)Analysing the status of an AI model with regard to the GDPR\.Note:Accessed 12 February 2026External Links:[Link](https://cnil.fr/fr/node/167980)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p2.1)\.
- S\. U\. Dar, M\. Seyfarth, I\. Ayx, T\. Papavassiliu, S\. O\. Schoenberg, R\. M\. Siepmann, F\. C\. Laqua, J\. Kahmann, N\. Frey, B\. Baeßler, and S\. Foersch \(2025\)Unconditional latent diffusion models memorize patient imaging data\.Nature Biomedical Engineering\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I1.i1.p1.1)\.
- H\. A\. Davis, D\. Kerkman, A\. A\. Hoberg, M\. Countryman, W\. Beaver, K\. Bybee, J\. M\. Blum, and B\. M\. Knosp \(2025\)Establishing data governance for sharing and access to real\-world data: a case study\.JAMIA Open8\(3\),pp\. ooaf041\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- J\. De Bruyne and C\. Van Leenhove \(2022\)Tort law and damage caused by ai systems\.pp\. 395–445\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p2.1)\.
- C\. Dwork and A\. Roth \(2014\)The algorithmic foundations of differential privacy\.Foundations and Trends in Theoretical Computer Science9\(3–4\),pp\. 211–407\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- European Commission \(2021\)Assessment of the eu member states’ rules on health data in the light of the gdpr\.Publications Office of the European Union,Luxembourg\.External Links:[Document](https://dx.doi.org/10.2818/546193),[Link](https://doi.org/10.2818/546193)Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- European Commission \(2026a\)European artificial intelligence board \(AI Board\)\.Note:Shaping Europe’s Digital FutureAccessed 12 February 2026External Links:[Link](https://digital-strategy.ec.europa.eu/en/policies/ai-board)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- European Commission \(2026b\)Medical Device Coordination Group \(MDCG\)\.Note:Register of Commission Expert GroupsAccessed 12 February 2026External Links:[Link](https://ec.europa.eu/transparency/expert-groups-register/)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- European Data Protection Board and European Data Protection Supervisor \(2026\)EDPB–edps joint opinion 2/2026 on the proposal for a regulation as regards the simplification of the digital legislative framework \(digital omnibus\)\.Note:Published 11 February 2026; accessed 26 February 2026External Links:[Link](https://www.edpb.europa.eu/system/files/2026-02/edpb_edps_jointopinion_202602_digitalomnibus_en.pdf)Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- European Federation of Data Protection Officers \(EFDPO\) \(2021\)Comments on the european data protection board’s guidelines 01/2021 on examples regarding data breach notification\.Note:Public consultation responseAccessed 12 February 2026External Links:[Link](https://www.edpb.europa.eu/system/files/documents/webform/public_consultation_reply/efdpo_position_examples_data_breaches-final.pdf)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- European Parliament and Council of the European Union \(2016\)Regulation \(EU\) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/EC \(general data protection regulation\)\.Official Journal of the European UnionL119,pp\. 1–88\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- European Parliament and Council of the European Union \(2022\)Regulation \(EU\) 2022/868 of the european parliament and of the council of 30 may 2022 on european data governance and amending regulation \(EU\) 2018/1724 \(data governance act\)\.Vol\.L 152\.Note:Official Journal of the European UnionAccessed 12 February 2026External Links:[Link](https://eur-lex.europa.eu/eli/reg/2022/868/oj/eng)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- European Parliament and Council of the European Union \(2023\)Regulation \(EU\) 2023/2854 of the european parliament and of the council of 13 december 2023 on harmonised rules on fair access to and use of data and amending regulation \(EU\) 2017/2394 and directive \(EU\) 2020/1828 \(data act\)\.Note:Official Journal of the European UnionAccessed 12 February 2026External Links:[Link](https://eur-lex.europa.eu/eli/reg/2023/2854/oj/eng)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- European Parliament and Council of the European Union \(2025\)Regulation \(EU\) 2025/327 of the european parliament and of the council of 11 february 2025 on the european health data space and amending directive 2011/24/EU and regulation \(EU\) 2024/2847\.Note:Official Journal of the European UnionAccessed 12 February 2026External Links:[Link](https://eur-lex.europa.eu/eli/reg/2025/327/oj/eng)Cited by:[Appendix B](https://arxiv.org/html/2608.07705#Sx3.p3.1)\.
- G\. R\. Ghosal, P\. Maini, and A\. Raghunathan \(2025\)Memorization sinks: isolating memorization during llm training\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 19307–19326\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- Google Research \(2022\)Med\-PaLM: a medical large language model\.Note:[https://sites\.research\.google/med\-palm/](https://sites.research.google/med-palm/)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1)\.
- X\. Gu, C\. Du, T\. Pang, C\. Li, M\. Lin, and Y\. Wang \(2023\)On memorization in diffusion models\.arXiv preprint arXiv:2310\.02664\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p3.1)\.
- L\. L\. Guo, J\. Fries, E\. Steinberg, S\. L\. Fleming, K\. Morse, C\. Aftandilian, J\. Posada, N\. Shah, and L\. Sung \(2024\)A multi\-center study on the adaptability of a shared foundation model for electronic health records\.npj Digital Medicine7,pp\. 171\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1),[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- L\. L\. Guo, E\. Steinberg, S\. L\. Fleming, J\. Posada, J\. Lemmon, S\. R\. Pfohl, N\. Shah, J\. Fries, and L\. Sung \(2023\)EHR foundation models improve robustness in the presence of temporal distribution shift\.Scientific Reports13\(1\),pp\. 3767\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1)\.
- V\. Gupta, B\. S\. Erdal, C\. Ramirez, R\. Floca, L\. Jackson, B\. Genereaux, S\. Bryson, C\. P\. Bridge, J\. Kleesiek, F\. Nensa, R\. Braren, K\. Younis, T\. Penzkofer, A\. M\. Bucher, M\. M\. Qin, G\. Bae, H\. Lee, M\. J\. Cardoso, S\. Ourselin, E\. Kerfoot, R\. Choudhury, R\. D\. White, T\. Cook, D\. Bericat, M\. Lungren, R\. Haukioja, and H\. Shuaib \(2024\)Current state of community\-driven radiological ai deployment in medical imaging\.JMIR AI3,pp\. e55833\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- S\. Hallaj, A\. Heinke, F\. G\. P\. Kalaw, N\. Gim, M\. Blazes, J\. Owen, E\. Dysinger, E\. S\. Benton, B\. A\. Cordier, N\. G\. Evans,et al\.\(2026\)Navigating open data sharing and privacy in the age of clinical ai research: from reidentification to pseudo\-reidentification\.EClinicalMedicine91\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I2.i1.p1.1)\.
- T\. Han, L\. C\. Adams, J\. Papaioannou, P\. Grundmann, T\. Oberhauser, A\. Figueroa, A\. Löser, D\. Truhn, and K\. K\. Bressem \(2023\)MedAlpaca—an open\-source collection of medical conversational AI models and training data\.arXiv preprint arXiv:2304\.08247\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1)\.
- Y\. He, F\. Huang, X\. Jiang, Y\. Nie, M\. Wang, J\. Wang, and H\. Chen \(2025\)Foundation model for advancing healthcare: challenges, opportunities and future directions\.IEEE Reviews in Biomedical Engineering18,pp\. 172–191\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- N\. Holzenberger and W\. Maxwell \(2025\)A quantitative approach to the GDPR’s anonymisation and “appropriate technical and organisational measures” tests\.Computer Law & Security Review59,pp\. 106173\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p6.1)\.
- H\. Hu, Z\. Salcic, L\. Sun, G\. Dobbie, and X\. Zhang \(2021\)Source inference attacks in federated learning\.In2021 IEEE International Conference on Data Mining \(ICDM\),pp\. 1102–1107\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p3.1)\.
- International AI Safety Report \(2026\)Second key update: technical safeguards and risk management\.Note:Accessed 26 February 2026External Links:[Link](https://internationalaisafetyreport.org/publication/second-key-update-technical-safeguards-and-risk-management)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p3.1)\.
- W\. Jeong, H\. J\. Kim, Z\. Deng, Y\. Shen, A\. I\. Aviles\-Rivero, and S\. Zhang \(Eds\.\) \(2026\)Foundation models for general medical ai: third international workshop, medagi 2025, held in conjunction with miccai 2025, daejeon, south korea, september 27, 2025, proceedings\.Lecture Notes in Computer Science, Vol\.16112,Springer Cham\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1),[1st item](https://arxiv.org/html/2608.07705#S3.I2.i1.p1.1)\.
- H\. Jun, Y\. Tanaka, S\. Johri, S\. Y\. Camp, E\. L\. Bao, F\. L\. F\. Carvalho, D\. Y\. Gui, A\. C\. Jordan, C\. Labaki, S\. D\. Martin, M\. Nagy, T\. A\. O’Meara, T\. Pappa, E\. M\. Pimenta, E\. Saad, D\. D\. Yang, R\. Gillani, A\. K\. Tewari, B\. Reardon, and E\. Van Allen \(2026\)A context\-augmented large language model for accurate precision oncology medicine recommendations\.Cancer Cell\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p2.1)\.
- M\. Jungkunz, A\. Köngeter, K\. Mehlis, E\. C\. Winkler, and C\. Schickhardt \(2021\)Secondary use of clinical data in data\-gathering, non\-interventional research or learning activities: definition, types, and a framework for risk assessment\.Journal of Medical Internet Research23,pp\. e26631\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- K\. L\. Kehl, J\. Jee, K\. Pichotta, M\. A\. Paul, P\. Trukhanov, C\. Fong, M\. Waters, Z\. Bakouny, W\. Xu, T\. K\. Choueiri, C\. Nichols, D\. Schrag, and N\. Schultz \(2024\)Shareable artificial intelligence to extract cancer outcomes from electronic health records for precision oncology research\.Nature Communications15,pp\. 9787\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I2.i1.p1.1)\.
- U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. Lewis \(2019\)Generalization through memorization: nearest neighbor language models\.arXiv preprint arXiv:1911\.00172\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- B\. Kulynych, M\. Yaghini, G\. Cherubin, M\. Veale, and C\. Troncoso \(2022\)Disparate vulnerability to membership inference attacks\.Proceedings on Privacy Enhancing Technologies2022\(1\),pp\. 460–480\.External Links:[Document](https://dx.doi.org/10.2478/popets-2022-0023)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p3.1)\.
- J\. Li, A\. D\. Aguirre, V\. Moura, J\. Jin, C\. Liu, L\. Zhong, C\. Sun, G\. D\. Clifford, M\. B\. Westover, and S\. Hong \(2025\)An electrocardiogram foundation model built on over 10 million recordings\.NEJM AI2\(7\),pp\. AIoa2401033\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- K\. Li, Q\. Wang, Y\. Wang, F\. Li, J\. Liu, B\. Han, and J\. Zhou \(2026\)Llm unlearning with llm beliefs\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 101587–101622\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p3.1)\.
- S\. Li, L\. Yao, L\. Zhang, and Y\. Li \(2024\)Safety layers in aligned large language models: the key to llm security\.arXiv preprint arXiv:2408\.17003\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- S\. M\. McKinney, M\. Sieniek, V\. Godbole, J\. Godwin, N\. Antropova, H\. Ashrafian, T\. Back, M\. Chesus, G\. S\. Corrado, A\. Darzi, M\. Etemadi, F\. Garcia\-Vicente, F\. J\. Gilbert, M\. Halling\-Brown, D\. Hassabis, S\. Jansen, A\. Karthikesalingam, C\. J\. Kelly, D\. King, J\. R\. Ledsam, D\. Melnick, H\. Mostofi, L\. Peng, J\. J\. Reicher, B\. Romera\-Paredes, R\. Sidebottom, M\. Suleyman, D\. Tse, K\. C\. Young, J\. De Fauw, and S\. Shetty \(2020\)International evaluation of an ai system for breast cancer screening\.Nature577,pp\. 89–94\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- H\. B\. McMahan, D\. Ramage, K\. Talwar, and L\. Zhang \(2018\)Learning differentially private recurrent language models\.International Conference on Learning Representations \(ICLR\)\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- M\. Meeus, L\. Wutschitz, S\. Zanella\-Béguelin, S\. Tople, and R\. Shokri \(2025\)The canary’s echo: auditing privacy risks of llm\-generated synthetic text\.External Links:2502\.14921,[Document](https://dx.doi.org/10.48550/arXiv.2502.14921),[Link](https://arxiv.org/abs/2502.14921)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- M\. Moor, O\. Banerjee, Z\. S\. H\. Abad, H\. M\. Krumholz, J\. Leskovec, E\. J\. Topol, and P\. Rajpurkar \(2023\)Foundation models for generalist medical artificial intelligence\.Nature616,pp\. 259–265\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- J\. Morley, L\. Murphy, A\. Mishra, I\. Joshi, and K\. Karpathakis \(2022\)Governing data and artificial intelligence for health care: developing an international understanding\.JMIR Formative Research6\(1\),pp\. e31623\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1),[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p1.1)\.
- G\. Noto La Diega and L\. C\. Bezerra \(2024\)Can there be responsible ai without ai liability? incentivizing generative ai safety through ex\-post tort liability under the eu ai liability directive\.International Journal of Law and Information Technology32\(1\),pp\. eaae021\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p2.1)\.
- C\. Novelli, F\. Casolari, P\. Hacker, G\. Spedicato, and L\. Floridi \(2024\)Generative AI in EU law: liability, privacy, intellectual property, and cybersecurity\.Computer Law & Security Review55,pp\. 106066\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- OpenAI \(2026\)Introducing ChatGPT Health\.Note:[https://openai\.com/index/introducing\-chatgpt\-health/](https://openai.com/index/introducing-chatgpt-health/)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1)\.
- K\. Packhäuser, D\. Schaub, T\. Huber, M\. Pfeiffer, G\. Schmidt, J\. N\. Kather, D\. Truhn, P\. Bruners, T\. Penzkofer, D\. Pinto dos Santos, K\. Maier\-Hein, and B\. Baeßler \(2022\)Deep learning\-based patient re\-identification is able to exploit the biometric nature of medical chest x\-ray data\.Scientific Reports12,pp\. 1–13\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I2.i1.p1.1)\.
- A\. Panda, X\. Tang, C\. Choquette\-Choo, M\. Nasr, and P\. Mittal \(2025\)Privacy auditing of large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 92555–92571\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p3.1)\.
- N\. Papernot, P\. McDaniel, and I\. Goodfellow \(2016\)Transferability in machine learning: from phenomena to black\-box attacks using adversarial samples\.arXiv preprint arXiv:1605\.07277\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx3.p1.1)\.
- E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving \(2022\)Red teaming language models with language models\.External Links:2202\.03286,[Document](https://dx.doi.org/10.48550/arXiv.2202.03286),[Link](https://arxiv.org/abs/2202.03286)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- H\. Pouransari, D\. Grangier, C\. Thomas, M\. Kirchhof, and O\. Tuzel \(2026\)Pretraining with hierarchical memories: separating long\-tail and common knowledge\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- A\. Purpura, S\. Wadhwa, J\. Zymet, A\. Gupta, A\. Luo, M\. K\. Rad, S\. Shinde, and M\. S\. Sorower \(2025\)Building safe genai applications: an end\-to\-end overview of red teaming for large language models\.InProceedings of the 5th Workshop on Trustworthy Natural Language Processing \(TrustNLP\),pp\. 335–350\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.trustnlp-1.24),[Link](https://aclanthology.org/2025.trustnlp-1.24)Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- R\. Raab, A\. Küderle, A\. Zakreuskaya, A\. D\. Stern, J\. Klucken, G\. Kaissis, D\. Rueckert, S\. Boll, R\. Eils, H\. Wagener,et al\.\(2023\)Federated electronic health records for the european health data space\.The Lancet Digital Health5\(11\),pp\. e840–e847\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p2.1)\.
- P\. Renc, Y\. Jia, A\. E\. Samir, J\. Was, Q\. Li, D\. W\. Bates, and A\. Sitek \(2024\)Zero shot health trajectory prediction using transformer\.npj Digital Medicine7,pp\. 256\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1),[§2](https://arxiv.org/html/2608.07705#S2.p1.1),[§2](https://arxiv.org/html/2608.07705#S2.p2.1)\.
- A\. R\. Sarkar, Y\. Chuang, N\. Mohammed, and X\. Jiang \(2024\)De\-identification is not enough: a comparison between de\-identified and synthetic clinical notes\.Scientific Reports14\(1\),pp\. 29669\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I1.i1.p1.1)\.
- A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau, J\. Chen, F\. Mahvar, L\. Yatziv, T\. Chen, B\. Sterling, S\. A\. Baby, S\. M\. Baby, J\. Lai, S\. Schmidgall, L\. Yang, K\. Chen, P\. Bjornsson, S\. Reddy, R\. Brush, K\. Philbrick, M\. Asiedu, I\. Mezerreg, H\. Hu, H\. Yang, R\. Tiwari, S\. Jansen, P\. Singh, Y\. Liu, S\. Azizi, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Riviere, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Buchatskaya, J\. Alayrac, D\. Lepikhin, V\. Feinberg, S\. Borgeaud, A\. Andreev, C\. Hardin, R\. Dadashi, L\. Hussenot, A\. Joulin, O\. Bachem, Y\. Matias, K\. Chou, A\. Hassidim, K\. Goel, C\. Farabet, J\. Barral, T\. Warkentin, J\. Shlens, D\. Fleet, V\. Cotruta, O\. Sanseviero, G\. Martins, P\. Kirk, A\. Rao, S\. Shetty, D\. F\. Steiner, C\. Kirmizibayrak, R\. Pilgrim, D\. Golden, and L\. Yang \(2025\)MedGemma technical report\.arXiv preprint\.Note:arXiv:2507\.05201Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1),[§2](https://arxiv.org/html/2608.07705#S2.p3.1)\.
- J\. A\. R\. Smit, M\. Mostert, R\. van der Graaf, D\. E\. Grobbee, and J\. J\. M\. van Delden \(2024\)Specific measures for data\-intensive health research without consent: a systematic review of soft law instruments and academic literature\.European Journal of Human Genetics32\(1\),pp\. 21–30\.External Links:[Document](https://dx.doi.org/10.1038/s41431-023-01457-2),[Link](https://doi.org/10.1038/s41431-023-01457-2)Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- K\. Spector\-Bagdady, R\. D\. Pentz, S\. Joffe, G\. E\. Henderson, S\. L\. R\. Kardia, J\. Bollinger, S\. A\. Kraft, D\. McGraw, S\. B\. Trinidad, B\. S\. Wilfond, N\. H\. Shah, R\. Platt, and D\. M\. Roden \(2023\)Principles for health information collection, sharing, and use: a policy statement from the american heart association\.Circulation148,pp\. 1061–1069\.Cited by:[§4](https://arxiv.org/html/2608.07705#S4.p6.1)\.
- T\. Stadler, B\. Oprisanu, and C\. Troncoso \(2022\)Synthetic data—anonymisation groundhog day\.In31st USENIX Security Symposium \(USENIX Security 22\),pp\. 1451–1468\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I1.i1.p1.1),[1st item](https://arxiv.org/html/2608.07705#S3.I2.i1.p1.1)\.
- K\. Sun, S\. Xue, F\. Sun, H\. Sun, Y\. Luo, L\. Wang, S\. Wang, N\. Guo, L\. Liu, T\. Zhao, X\. Wang, L\. Yang, S\. Jin, J\. Yan, and J\. Dong \(2025\)Medical multimodal foundation models in clinical diagnosis and treatment: applications, challenges, and future directions\.Artificial Intelligence in Medicine170,pp\. 103265\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- Y\. C\. Tham, J\. H\. L\. Goh, and D\. Zur \(2025\)Building the world’s first truly global medical foundation model\.Nature Medicine31,pp\. 3580–3585\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- R\. Thapa, M\. R\. Kjaer, B\. He, I\. Covert, I\. Moore, U\. Hanif, G\. Ganjoo, M\. B\. Westover, P\. Jennum, A\. Brink\-Kjaer, E\. Mignot, and J\. Zou \(2026\)A multimodal sleep foundation model for disease prediction\.Nature Medicine32,pp\. 752–762\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- K\. Tirumala, A\. H\. Markosyan, L\. Zettlemoyer, and A\. Aghajanyan \(2022\)Memorization without overfitting: analyzing the training dynamics of large language models\.Advances in Neural Information Processing Systems35,pp\. 38274–38290\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- N\. Tomašev, X\. Glorot, J\. W\. Rae, M\. Zielinski, H\. Askham, A\. Saraiva, A\. Mottram, C\. Meyer, S\. Ravuri, I\. Protsyuk, A\. Connell, C\. O\. Hughes, A\. Karthikesalingam, J\. Cornebise, H\. Montgomery, G\. Rees, C\. Laing, C\. R\. Baker, K\. Peterson, R\. Reeves, D\. Hassabis, D\. King, M\. Suleyman, T\. Back, C\. Nielson, J\. R\. Ledsam, and S\. Mohamed \(2019\)A clinically applicable approach to continuous prediction of future acute kidney injury\.Nature572,pp\. 116–119\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p1.1)\.
- S\. Tonekaboni, L\. Stempfle, A\. Fallahpour, W\. Gerych, and M\. Ghassemi \(2025\)An investigation of memorization risk in healthcare foundation models\.arXiv preprint arXiv:2510\.12950\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx2.p2.1)\.
- U\.S\. Department of Health and Human Services, Office for Civil Rights \(2025\)Breach report\.Note:OCR PortalAccessed 26 February 2026External Links:[Link](https://ocrportal.hhs.gov/ocr/breach/breach_report.jsf)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- U\.S\. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation \(1996\)Health insurance portability and accountability act of 1996\.Note:Accessed 26 February 2026External Links:[Link](https://aspe.hhs.gov/reports/health-insurance-portability-accountability-act-1996)Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p3.1)\.
- A\. Wang, D\. E\. Ho, and S\. Koyejo \(2025\)The inadequacy of offline llm evaluations: a need to account for personalization in model behavior\.arXiv preprint arXiv:2509\.19364\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p3.1)\.
- C\. Wu, X\. Zhang, Y\. Zhang, H\. Hui, Y\. Wang, and W\. Xie \(2025\)Towards generalist foundation model for radiology by leveraging web\-scale 2D&3D medical data\.Nature Communications16\(1\),pp\. 7866\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- Q\. Xie, Q\. Chen, A\. Chen, C\. Peng, Y\. Hu, F\. Lin, X\. Peng, J\. Huang, J\. Zhang, V\. Keloth, X\. Zhou, L\. Qian, H\. He, D\. Shung, L\. Ohno\-Machado, Y\. Wu, H\. Xu, and J\. Bian \(2025\)Medical foundation large language models for comprehensive text analysis and beyond\.npj Digital Medicine8\(1\),pp\. 141\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- Q\. Xie, Q\. Chen, A\. Chen, C\. Peng, Y\. Hu, F\. Lin, X\. Peng, J\. Huang, J\. Zhang, V\. Keloth, and X\. Zhou \(2024\)Me\-LLaMA: foundation large language models for medical applications\.Research Square\.Cited by:[§1](https://arxiv.org/html/2608.07705#S1.p2.1)\.
- Q\. Yang, C\. Lepore, J\. Eynard, and R\. Laborde \(2025\)From theory to practice: data minimisation and technical review of verifiable credentials under the gdpr\.Computer Law & Security Review57,pp\. 106138\.Cited by:[§6](https://arxiv.org/html/2608.07705#S6.SSx1.p1.1)\.
- X\. Yang, A\. Chen, N\. PourNejatian, H\. C\. Shin, K\. E\. Smith, C\. Parisien, C\. Compas, C\. Martin, A\. B\. Costa, M\. G\. Flores, Y\. Zhang, T\. Magoc, C\. A\. Harle, G\. Lipori, D\. A\. Mitchell, W\. R\. Hogan, E\. A\. Shenkman, J\. Bian, and Y\. Wu \(2022\)A large language model for electronic health records\.npj Digital Medicine5\(1\),pp\. 194\.Cited by:[§2](https://arxiv.org/html/2608.07705#S2.p1.1)\.
- J\. Yoon, M\. Mizrahi, N\. F\. Ghalaty, T\. Jarvinen, A\. S\. Ravi, P\. Brune, F\. Kong, D\. Anderson, G\. Lee, A\. Meir, F\. Bandukwala, E\. Kanal, S\. Ö\. Arık, and T\. Pfister \(2023\)EHR\-safe: generating high\-fidelity and privacy\-preserving synthetic electronic health records\.npj Digital Medicine6\(1\),pp\. 141\.Cited by:[1st item](https://arxiv.org/html/2608.07705#S3.I1.i1.p1.1)\.

## Appendix A

As a reference for whether de\-identified data is “identifiable”, we can look at the recent jurisprudence by the European Court of Justice \(Case C\-413/23\)\. In that case, the Court clarified that pseudonymised data is not automatically “personal data” for all parties\. Rather, identifiability must be assessed relative to the specific actor and the technical and organisational measures in place\. If a recipient does not possess, and is not reasonably likely to obtain, the additional information required for re\-identification—because such access is technically prevented or legally prohibited—the data may, for that recipient, amount to effectively anonymous data\. This reflects a modified relative interpretation of identifiability, under which the classification of data depends on the concrete capabilities and legal position of the processing party\. While this approach increases contextual precision, it has also attracted academic criticism for potentially fragmenting the concept of personal data across actors\.

## Appendix B

EU AI ACT and MDR: relevant guidelines and articles

1. 1\.High\-risk classification under the AI Act - •Legal basis: Annex I Section A No\. 11 AIA\. - •AI systems that are safety components of products, or are themselves products, subject to third\-party conformity assessment under sectoral EU legislation \(including the MDR\) are automatically classified as high\-risk\. - •This includes medical device software falling within the scope of the MDR where a notified body is involved\.
2. 2\.Substantive obligations for high\-risk AI systems \(AI Act\) - •Art\. 9 – Risk management system: Continuous, documented risk identification, analysis, evaluation, and mitigation throughout the lifecycle\. - •Art\. 10 – Data and data governance: Requirements on training, validation, and testing data quality, relevance, representativeness, and bias mitigation\. - •Art\. 11 & Annex IV – Technical documentation: Comprehensive documentation enabling assessment of compliance, including system description, intended purpose, design specifications, and risk management measures\. - •Art\. 14 – Human oversight: Measures ensuring effective human control and the ability to intervene\. - •Art\. 15 – Accuracy, robustness, and cybersecurity: Performance requirements proportionate to the intended purpose and risk profile\.
3. 3\.Qualification as medical device software under the MDR - •Legal basis: Rule 11, Annex VIII MDR\. - •Software intended to provide information for diagnostic or therapeutic purposes qualifies as medical device software\. - •Risk classification \(Class I, IIa, IIb, III\) depends on the potential impact of the software’s output on patient management and health outcomes\. - •Class IIa or higher generally requires third\-party conformity assessment by a notified body\.
4. 4\.Interaction between MDR and AIA - •Where medical device software is subject to third\-party conformity assessment under the MDR, it is automatically classified as high\-risk under Annex I Section A AIA\. - •Conformity assessments may be aligned or integrated, but the substantive requirements of both regimes apply cumulatively\.

Other elements of the EU regulatory ecosystem include the Data Act\[European Parliament and Council of the European Union,[2023](https://arxiv.org/html/2608.07705#bib.bib37)\]37, the Data Governance Act\[European Parliament and Council of the European Union,[2022](https://arxiv.org/html/2608.07705#bib.bib38)\], and the European Health Data Space \(EHDS\) Regulation\[European Parliament and Council of the European Union,[2025](https://arxiv.org/html/2608.07705#bib.bib39)\]which primarily structure data access and reuse frameworks and indirectly influence patient privacy\. As the EU framework for digital regulation is still emerging, the practical interpretation of many sources is driven by guidelines, codes of practice, and supporting documents produced by the European Commission and various stakeholder groups, such as the European Data Protection Board \(EDPB\)\[Barberá,[2025](https://arxiv.org/html/2608.07705#bib.bib26)\], European Artificial Intelligence Board \(AIB\)\[European Commission,[2026a](https://arxiv.org/html/2608.07705#bib.bib40)\], Medical Device Coordination Group \(MDCG\)\[European Commission,[2026b](https://arxiv.org/html/2608.07705#bib.bib55)\], and the European Association of Data Protection Professionals \(EADPP\)\[European Federation of Data Protection Officers \(EFDPO\),[2021](https://arxiv.org/html/2608.07705#bib.bib56)\]\.