Demystifying the Privacy-Utility Trade-off in LLM Interactions

# Demystifying the Privacy-Utility Trade-off in LLM Interactions
Source: [https://arxiv.org/html/2609.10992](https://arxiv.org/html/2609.10992)
Zhanxu XieJunjie YuTong ZhuLijun Li\\correspondingWenliang Chen\\corresponding

###### Abstract

The integration of Large Language Models into daily tasks relies on context\-rich instructions, inevitably exposing sensitive user information\. Current privacy\-preserving methods typically employ context\-agnostic static rules, causing severe utility degradation\. However, the specific mechanisms governing how sanitization impacts downstream performance remain largely underexplored\. To address this, we conduct a systematic analysis to deconstruct the privacy\-utility trade\-off, uncovering three underlying mechanisms: \(1\)Context\-Dependent Utility, which first establishes when to sanitize by revealing that data value shifts from critical constraints to dispensable noise based on user intent; \(2\)Strategic Adaptation, which subsequently determines how to sanitize by dictating that the choice between removal and replacement depends on the task’s reliance on factual integrity versus structural coherence; and \(3\)Combinatorial Interplay, which finally extends the protection scope by demonstrating that attributes form a semantic web of synergistic dependencies or antagonistic redundancies\. Guided by these insights, we introduce an intent\-driven local protection framework\. By distilling a lightweight modelVeilmind\-4Bto drive a dynamic extraction\-sanitization\-restoration pipeline, our approach reaches a low\-leakage privacy point while preserving substantially higher response utility than existing privacy\-oriented baselines, advancing the privacy\-utility trade\-off toward the Pareto frontier\.

1Soochow University,2Beihang University,3Suzhou City University,

4Shanghai Key Lab of Intelligent Information Processing,5Shanghai AI Lab,

\{zhliu0106, wlchen\}@stu\.suda\.edu\.cn, xiezhanxu@buaa\.edu\.cn, jjyu@szcu\.edu\.cn, zhutong@pjlab\.org\.cn, 4065156@qq\.com

## 1Introduction

Large Language Models \(LLMs\) have evolved into intelligent agents that seamlessly integrate into diverse user workflows\([Brown et al\. 2020](https://arxiv.org/html/2609.10992#bib.bib5);[Xi et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib4)\)\. To elicit high\-quality responses, users must inevitably provide granular instructions rich in personal context and specific constraints\([Salemi et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib7);[Ouyang et al\. 2022](https://arxiv.org/html/2609.10992#bib.bib8)\)\. Such detailed disclosure serves as the cornerstone for agents to accurately interpret intent and deliver personalized assistance\([Wei et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib6)\)\.

However, high performance comes at a cost\. To elicit precise assistance, users must disclose granular details that inevitably expose sensitive personal attributes\. Crucially, data sensitivity is not static\. It depends strictly on the user’s intent and the specific task context\. This aligns with the theory of contextual integrity\([Nissenbaum 2004](https://arxiv.org/html/2609.10992#bib.bib9);[Nissenbaum 2009](https://arxiv.org/html/2609.10992#bib.bib10)\)\. In practice, even seemingly innocuous prompts can enable the reconstruction of detailed user profiles\([Staab et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib11)\)\. As depicted in Figure[1](https://arxiv.org/html/2609.10992#S1.F1), a routine task like drafting an insurance appeal necessitates the disclosure of clinical diagnoses and financial distress\. Once transmitted to untrusted servers, this data risks being mined for detailed profiling and subsequently exploited for targeted real\-world abuse\([Carlini et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib13);[Neel and Chang 2023](https://arxiv.org/html/2609.10992#bib.bib12);[Wang et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib14)\)\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/privacy_intro_v2.png)Figure 1:An illustrative example demonstrating how a user prompt can inadvertently disclose sensitive personal attributes, which may subsequently be exploited for targeted real\-world abuse\.To mitigate these risks, prior research has primarily relied on static sanitization rules or generic heuristics\([Lison et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib39);[Edemacu and Wu 2025](https://arxiv.org/html/2609.10992#bib.bib15)\)\. While effective at reducing privacy risks, they frequently compromise the utility of the response\([Feyisetan et al\. 2020](https://arxiv.org/html/2609.10992#bib.bib16);[Chowdhury et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib3)\)\. The critical limitation lies in their indiscriminate treatment of sensitive attributes\. Such methods fail to evaluate the marginal contribution of specific details relative to the user’s goal, often stripping away essential context alongside non\-essential noise\([Mireshghallah et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib17)\)\. Consequently, the field lacks a granular understanding of how varying degrees of sanitization impact model performance, leaving the underlying mechanisms ofthe privacy\-utility trade\-offlargely unexplored\.

To address this gap, we conduct a systematic empirical analysis to deconstruct the privacy\-utility trade\-off\. Unlike prior black\-box approaches, we move beyond static heuristics to map the underlying decision boundaries, answering three fundamental questions:

- •When to Sanitize?We reveal that the functional value of sensitive information is contingent on the task context\. We observe distinct dynamics ranging from functional coupling where attributes act as critical constraints to informational redundancy where they serve as dispensable noise\. Consequently, decisions on necessity must be grounded in this contextual utility\.
- •How to Sanitize?We demonstrate that optimal protection hinges on prioritizing either factual integrity or structural coherence\. Objective tasks demand removal to prevent false premises while interactive scenarios require replacement to serve as conversational anchors\. Thus, the choice of strategy is strictly dictated by the task’s reliance on factual versus structural validity\.
- •What Scope to Sanitize?We demonstrate that sensitive attributes do not function in isolation but form a complex semantic web\. This structure creates either synergistic bundles essential for coherence or antagonistic redundancy capable of leaking information\. Consequently, protection strategies must transcend individual evaluation to address the scope of these combinatorial dependencies\.

Guided by these insights, we propose an intent\-driven local protection framework\. To endow a lightweight local model with the advanced reasoning capabilities of state\-of\-the\-art systems, we employ knowledge distillation to specialize it for privacy\-centric tasks\. This model drives a dynamic three\-stage pipeline: extraction, strategic sanitization, and post\-hoc context restoration\. Crucially, to accommodate varying user tolerances, our framework provides flexible control viaUtility PriorityandPrivacy Prioritymodes\. Experimental results show that our framework advances the privacy\-utility Pareto frontier by reaching a low\-leakage privacy point while preserving substantially stronger utility than existing privacy\-oriented baselines\.

## 2Related Work

##### LLM Privacy\-Preserving Techniques\.

Extensive research addresses the risk of LLMs memorizing sensitive information from their training corpora\([Carlini et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib13);[Liu et al\. 2024b](https://arxiv.org/html/2609.10992#bib.bib19)\)\. The most direct approach is data sanitization, which entails detecting and redacting personally identifiable information \(PII\) from datasets\([Lison et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib39);[Kandpal et al\. 2022](https://arxiv.org/html/2609.10992#bib.bib20)\)\. While heuristic\-based filtering and named entity recognition \(NER\) models\([Chen et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib21)\)can mitigate explicit leakage, they are prone to false negatives\([Lukas et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib22)\)and can disrupt linguistic coherence\. To provide formal privacy guarantees, researchers have alternatively integrated differential privacy \(DP\) into the training process\([Abadi et al\. 2016](https://arxiv.org/html/2609.10992#bib.bib23);[Li et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib24)\)\. Recent works attempt to scale these methods to LLMs\([Yu et al\. 2022](https://arxiv.org/html/2609.10992#bib.bib25);[Sinha et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib26)\)\. However, despite its theoretical robustness, DP often incurs a substantial "utility tax", where the injection of noise inevitably compromises the model’s capabilities\.

Complementing these preventative strategies, machine unlearning has emerged as a post\-hoc remediation technique, particularly to comply with regulations like the "Right to be Forgotten"\([Jang et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib27);[Liu et al\. 2024a](https://arxiv.org/html/2609.10992#bib.bib28);[Liu et al\. 2025b](https://arxiv.org/html/2609.10992#bib.bib18)\)\. The objective is to erase the influence of specific sensitive samples from a trained model without retraining\. While promising, it faces the challenge of catastrophic forgetting, where the removal of specific knowledge inadvertently degrades the model’s general capabilities\([Yao et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib29);[Xu et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib30)\)\.

##### Privacy Protection in User\-LLM Interactions\.

In contrast to training\-side defenses, inference\-phase protection focuses on safeguarding user input from third\-party providers\. Early attempts addressed this via representation perturbation or formal privacy mechanisms\. Techniques such as adding noise to input embeddings\([Feyisetan et al\. 2020](https://arxiv.org/html/2609.10992#bib.bib16);[Plant et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib31);[Meehan et al\. 2022](https://arxiv.org/html/2609.10992#bib.bib33);[Du et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib32)\)or employing differentially private decoding\([Majmudar et al\. 2022](https://arxiv.org/html/2609.10992#bib.bib34);[Zhang et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib35);[Zeng et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib36)\)aim to render inputs unreadable to humans or statistically secure\. However, these methods face significant practical hurdles\. Embedding\-based approaches are incompatible with widespread text\-only commercial APIs and remain vulnerable to inversion attacks\([Song and Raghunathan 2020](https://arxiv.org/html/2609.10992#bib.bib37)\)\. Similarly, applying DP noise directly to text generation often disrupts semantic integrity, rendering prompts unintelligible to the downstream model and severely degrading task performance\.

Consequently, research shifts toward direct text\-level sanitization\. While standard rule\-based or NER methods are computationally efficient, their context\-agnostic nature often necessitates the removal of task\-relevant details, leading to utility collapse\([Microsoft 2025](https://arxiv.org/html/2609.10992#bib.bib38);[Lison et al\. 2021](https://arxiv.org/html/2609.10992#bib.bib39)\)\. To mitigate this, recent works leverage LLMs for more flexible protection\. Approaches range from reformulating out\-of\-context information\([Ngong et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib2)\)to dynamically adjusting sanitization strength based on leakage risk\([Shen et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib40)\)\. Other strategies explore architectural or cryptographic solutions, such as delegating inference between local and remote models\([Li et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib1)\)or employing format\-preserving encryption for sensitive tokens\([Chowdhury et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib3)\)\. Although superior to rigid rules, these methods often rely on static heuristics\. Crucially, they overlook the combinatorial Interplay of sensitive attributes and fail to strategically adapt sanitization actions to user intent\. Consequently, this lack of granularity yields suboptimal privacy\-utility trade\-offs\.

## 3Deconstructing the Privacy\-Utility Trade\-off

### 3\.1Data Collection and Annotation

Rigorous empirical analysis requires a high\-quality dataset consisting of user prompts annotated with their inherent sensitive statements\. Our data collection pipeline integrates real\-world distributions with retrieval\-augmented synthesis to ensure both diversity and contextual depth\.

##### Real\-world and Synthetic Data\.

We adopt the ShareGPT\-X\([DeSULT 2025](https://arxiv.org/html/2609.10992#bib.bib41)\), LMSYS\-Chat\-1M\([Zheng et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib42)\)and WildChat\([Zhao et al\. 2024a](https://arxiv.org/html/2609.10992#bib.bib43)\)dataset to capture diverse real\-world user intents\. To ensure sufficient coverage of sensitive attributes, we augment this corpus with synthetic data\. Specifically, we inject personal profiles from the Nemotron\-Personas dataset\([Meyer and Corneil 2025](https://arxiv.org/html/2609.10992#bib.bib44)\)into user queries without sensitive information, creating samples that are both semantically coherent and rich in sensitive details\.

##### Annotation and Validation\.

We define asensitive statementas a discrete textual unit \(e\.g\., a clause\) that reveals specific personal attributes\. We implement an automated extraction pipeline using a large reasoning model \(LRM\) to identify these statements\. To guarantee label quality, we enforce a rigorous verification protocol\. Manual inspection of a random subset \(N=200N=200\) yields an error rate of 2\.5%, confirming the reliability of our automated pipeline\. The final dataset comprises 9,757 samples \(5,928 real\-world and 3,829 synthetic\)\. The supplementary material provides additional implementation details\.

### 3\.2Experimental Setup

To empirically quantify the marginal utility of sanitization, we design a controlled experimental framework\. Unlike prior black\-box approaches, we structure our analysis to isolate the effects of user intent, sanitization strategy, and attribute interaction\. The following subsections detail the data taxonomy and evaluation metrics operationalizing these dimensions\.

#### Data Selection and Taxonomy

To investigate combinatorial interplay while eliminating information volume as a confounding factor, we construct a controlled subset,𝒟multi\\mathcal\{D\}\_\{\\text\{multi\}\}\. We filter for prompts containing a fixed number of sensitive statements \(specifically,N=5N=5\), yielding 384 high\-density samples\. This standardization ensures that observed utility variations are driven by intent and content semantics rather than the quantity of disclosed attributes\.

We employgemini\-2\.5\-flash\([Gemini Team 2025](https://arxiv.org/html/2609.10992#bib.bib46)\)to annotate this subset along two orthogonal dimensions\. As detailed in the supplementary material, we categorize prompts into sixIntent Types, ranging from open\-endedCreation & Ideationto constrainedTask Execution\. Simultaneously, we classify sensitive statements into sevenPrivacy Typesaligned with standard PII definitions\. Figure[2](https://arxiv.org/html/2609.10992#S3.F2)visualizes the distribution across these categories\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/pie-2.png)\(a\)Intent Type
![Refer to caption](https://arxiv.org/html/2609.10992v1/pie-1.png)\(b\)Privacy Type
![Refer to caption](https://arxiv.org/html/2609.10992v1/count_heatmap.png)\(c\)Joint Distribution

Figure 2:Dataset statistics illustrating the marginal and joint distributions of intent types and privacy types\.
#### Evaluation Pipeline

We design a pipeline to quantify the marginal utility shift \(Δ​U\\Delta U\) resulting from privacy sanitization\. The pipeline consists of two core components: a response generator and a utility evaluator\.

##### Remote Model\.

We adoptgemini\-2\.5\-flash\([Gemini Team 2025](https://arxiv.org/html/2609.10992#bib.bib46)\)as the remote model to generate responses for original and sanitized prompts\. To ensure reproducibility, we configure the decoding parameters to deterministic values withtemperature=0andmax\_tokens=8192\.

##### Utility Metric\.

We leverageSkywork\-Reward\-V2\-Llama\-3\.1\-8B\([Liu et al\. 2025a](https://arxiv.org/html/2609.10992#bib.bib47)\)as the proxy for utility evaluation\. Given a prompt and a response, the model assigns a scalar quality scoreRR\. We quantify the utility impact as the difference in reward scores between the sanitized response and the original response\.

### 3\.3Context\-Dependent Utility: The Interplay of User Intent and Sensitive Information

We investigate how the removal of specific sensitive statements differentially impacts utility across varying user intents\. To address this, we conduct a fine\-grained ablation study on the subset𝒟multi\\mathcal\{D\}\_\{\\text\{multi\}\}\. We perform an ablation operation on each sample to remove one target statement while preserving the remaining context\. We employgemini\-2\.5\-flashto execute this targeted removal following the instruction template provided in the supplementary material\. We then generate responses for both original and sanitized prompts via the remote model\. We quantify the utility impact asΔ​R=Rs​a​n​i​t​i​z​e​d−Ro​r​i​g​i​n​a​l\\Delta R=R\_\{sanitized\}\-R\_\{original\}\. A negative value indicates performance degradation\. Figure[3](https://arxiv.org/html/2609.10992#S3.F3)visualizes the resulting utility sensitivity matrix\.

Based on the empirical evidence from Figure[3](https://arxiv.org/html/2609.10992#S3.F3), we summarize three findings:

- •Finding 1: Intent\-Dictated Utility Sensitivity\.User intents strictly dictate the functional value of sensitive context\. In constraint\-driven tasks, specific details define the solution space\. Consequently, removingSocial and Relational InformationduringTask Executioncauses severe performance degradation \(Δ​R=−14\.25\\Delta R=\-14\.25\)\. Conversely, logic\-driven tasks often treat personal attributes as orthogonal noise\. Notably, removingHealth and Wellnessdata inTask Executionyields a utility gain \(Δ​R=\+0\.19\\Delta R=\+0\.19\)\. This indicates that unrelated privacy leakage can distract the model from the core objective\.
- •Finding 2: Attribute\-Specific Functional Roles\.Sensitive attributes display distinct utility profiles rather than uniform importance\.Social and Relational Informationexhibits high context sensitivity\. It acts as a critical prerequisite for collaborative execution yet plays a negligible role inPersonalized Interaction\(Δ​R=−2\.07\\Delta R=\-2\.07\)\. In contrast,Health and Wellnessdata demonstrates high specificity\. It is vital for establishing empathy inPersonalized Interaction\(Δ​R=−5\.78\\Delta R=\-5\.78\) but serves as distractor noise inAnalysis & Reasoning\(Δ​R=\+0\.12\\Delta R=\+0\.12\)\.
- •Finding 3: Functional Coupling vs\. Informational Redundancy\.The interaction between user intent and sensitive content dictates the optimal protection strategy\. We observe functional coupling where privacy leakage is a prerequisite for utility\. This is evident in the strong dependency ofProblem SolvingonFinancial Information\. In contrast, informational redundancy occurs when sensitive attributes provide no marginal value\. The positive reward shift upon removingFinancial InformationfromInformation Acquisition\(Δ​R=\+0\.56\\Delta R=\+0\.56\) confirms that privacy preservation in such contexts incurs no utility cost\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/remove_reward_change_heatmap.png)Figure 3:The utility sensitivity matrix quantifying the impact of removing sensitive information across different user intents\.![Refer to caption](https://arxiv.org/html/2609.10992v1/replace_reward_change_heatmap.png)\(a\)Impact of Replacement Strategy
![Refer to caption](https://arxiv.org/html/2609.10992v1/comparison_diff_heatmap.png)\(b\)Strategic Differential

Figure 4:Comparative utility analysis of sanitization strategies: \(a\) Quantifies the marginal utility shift caused by substituting sensitive attributes with synthetic placeholders\. \(b\) Visualizes the strategic differential, where positive values \(blue\) indicate removal preserves more utility, while negative values \(red\) favor replacement\.
### 3\.4Strategic Adaptation: The Efficacy of Removal versus Replacement

Removal serves as the standard baseline for sanitization\. However, it often fractures the semantic coherence of the prompt\. To address this, we evaluate aReplacementstrategy, which substitutes sensitive attributes with synthetic placeholders using the supplementary template, and visualize the resulting utility impact in Figure[4\(a\)](https://arxiv.org/html/2609.10992#S3.F4.sf1)\. To rigorously compare the relative efficacy of these two approaches, we define the strategic differential asD=Δ​Rremove−Δ​RreplaceD=\\Delta R\_\{\\text\{remove\}\}\-\\Delta R\_\{\\text\{replace\}\}\. The distribution of this differential is presented in Figure[4\(b\)](https://arxiv.org/html/2609.10992#S3.F4.sf2), where positive values \(blue regions\) indicate that removal preserves more utility, while negative values \(red regions\) favor replacement\.

Our comparative analysis reveals two distinct patterns driven by the functional role of the information:

- •Finding 1: Factual Integrity and the Risk of False Premises\.For tasks grounded in objective execution, accuracy is paramount\. In these scenarios, providing a fabricated value via replacement is significantly more damaging than omitting the data through removal\. We observe that synthetic placeholders often function as false premises, causing the model to hallucinate solutions based on incorrect pathologies or constraints\. This is evident in the interaction betweenTask ExecutionandHealth Information, where the strategic differential peaks atD=\+14\.04D=\+14\.04\. Consequently, for logic\-driven tasks, silence is superior to noise\.
- •Finding 2: Structural Coherence and Narrative Anchoring\.Conversely, replacement strategy demonstrates a comparative advantage when the user seeks structural guidance or empathy rather than factual precision\. In these contexts, sensitive statements act as conversational anchors that maintain the dialogue flow\. For instance, inPersonalized Interaction, replacingHealthdata allows the model to preserve an empathetic tone, thereby outperforming removal \(D=−1\.64D=\-1\.64\)\. Similarly, inProblem Solving, using a dummyFinancialfigure \(D=−1\.33D=\-1\.33\) enables the model to demonstrate a correct calculation process\. In such cases, the structural validity of the response outweighs the factual accuracy of the input\.

### 3\.5Combinatorial Interplay: The Synergy and Antagonism of Sensitive Information

Sensitive statements rarely function in isolation; rather, they work in an interconnected manner where one attribute affects the utility of another\. To measure these interactions, we conduct pairwise removals on𝒟multi\\mathcal\{D\}\_\{\\text\{multi\}\}using the prompt template provided in the supplementary material\. For every pair of sensitive statements\(A,B\)\(A,B\), we remove both simultaneously and measure the utility shiftΔ​R​\(A,B\)\\Delta R\(A,B\)\. We define the interaction score \(IA,BI\_\{A,B\}\) as the difference between the joint impact and the sum of individual impacts:

IA,B=Δ​R​\(A,B\)−\(Δ​R​\(A\)\+Δ​R​\(B\)\)I\_\{A,B\}=\\Delta R\(A,B\)\-\(\\Delta R\(A\)\+\\Delta R\(B\)\)\(1\)A negative score \(I<0I<0\) indicates synergy, where removing both attributes together causes a utility drop significantly larger than the sum of removing them individually\. Conversely, a positive score \(I\>0I\>0\) indicates antagonism, implying that the attributes are redundant and provide overlapping information\. We visualize the global distribution of these effects in Figure[5](https://arxiv.org/html/2609.10992#S3.F5); the supplementary material provides the user\-intent breakdown\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/synergy_antagonism_overall_heatmap.png)Figure 5:The global interaction matrix illustrating the combinatorial interplay of sensitive statements\. Blank cells indicate no available data for that privacy type pair\.Our analysis yields three findings:

- •Finding 1: Narrative Cohesion Drives Synergy\.Strong synergistic effects emerge when two statements are logically interlinked to construct a coherent user story\. As evidenced by the deep blue regions in the global heatmap, we observe significant synergy betweenHealth and WellnessandInterests, Beliefs, and Opinions\(I=−6\.05I=\-6\.05\)\. These pairs often establish a causal link, such as a health condition motivating a specific lifestyle change\. Consequently, severing both links destroys the causal chain, leaving the model with no basis to infer the user’s underlying motivation\.
- •Finding 2: Informational Redundancy Drives Antagonism\.Conversely, antagonistic effects appear when statements provide overlapping signals\. In the global view,Behavioral DataandSocial Informationexhibit high antagonism \(I=\+4\.14I=\+4\.14\)\. For example, a social status like "Student" inherently implies behavioral patterns such as "Studying," rendering the explicit statement of behavior redundant\. In this case, removing only one attribute fails to protect privacy, as the model can reconstruct the missing context from its counterpart\.
- •Finding 3: User Intent Dictates Interaction Patterns\.The nature of interaction is strictly determined by user intent rather than being inherent to the data categories\. InTask Execution, we observe extreme antagonism where eliminating bothSocial InformationandInterestsyields a peak score ofI=\+47\.8I=\+47\.8\. This confirms that simultaneously removing unrelated noise significantly amplifies model focus\. Distinctly,Personalized Supportdemonstrates a selective anchoring effect, where the presence of criticalHealthcontext rendersProfessional Backgroundfunctionally redundant \(I=\+28\.2I=\+28\.2\)\.

## 4Framework Implementation and Evaluation

Section[3](https://arxiv.org/html/2609.10992#S3)established three key insights regarding the privacy\-utility trade\-off: context\-dependent utility, strategic adaptation, and combinatorial interplay\. Existing methods relying on static rules cannot address these complexities\. In this section, we propose an Intent\-Driven Local Protection Framework that directly applies these insights\. Instead of using rigid heuristics, we formulate our empirical findings into explicit reasoning instructions\. Through knowledge distillation, we embed this logic into a lightweight local model, teaching it to evaluate data sensitivity based on user intent\. This enables the framework to execute a precise three\-stage pipeline: Extraction, Sanitization, and Restoration\.

### 4\.1Architecture Design

The inference workflow of our framework consists of three sequential modules\.

##### Stage 1: Sensitive Information Extraction\.

Given a user promptPP, the local model first identifies and extracts the set of sensitive statements𝒮=\{s1,s2,…,sn\}\\mathcal\{S\}=\\\{s\_\{1\},s\_\{2\},\\dots,s\_\{n\}\\\}\. This process requires precise entity recognition and contextual understanding to capture both explicit PII and implicit attribute disclosure\.

##### Stage 2: Strategic Planning and Sanitization\.

This module constitutes the core decision\-making engine\. Guided by the principles derived in Section[3](https://arxiv.org/html/2609.10992#S3), the model evaluates each statementsi∈𝒮s\_\{i\}\\in\\mathcal\{S\}based on the user’s intent and the privacy category\. It generates a protection planπ\\pithat assigns an actionai∈\{Keep,Remove,Replace\}a\_\{i\}\\in\\\{\\textsc\{Keep\},\\textsc\{Remove\},\\textsc\{Replace\}\\\}to each statement\.

Simultaneously, the model executes this plan to transform the original promptPPinto a sanitized versionP′P^\{\\prime\}\.

P′,π=ℳlocal​\(P,𝒮\)P^\{\\prime\},\\pi=\\mathcal\{M\}\_\{\\text\{local\}\}\(P,\\mathcal\{S\}\)\(2\)Crucially, the decision logic accounts for the interaction effects between statements\. The model removes antagonistic pairs to prevent redundancy leakage and preserves synergistic pairs when necessary for task utility\.

##### Stage 3: Response Restoration\.

The sanitized promptP′P^\{\\prime\}is transmitted to the untrusted remote model, which returns a generic responseR′R^\{\\prime\}\. To ensure the final output remains personalized and relevant, our local model performs a post\-processing step\. It utilizes the stored planπ\\piand the original sensitive statements𝒮\\mathcal\{S\}to re\-inject necessary context intoR′R^\{\\prime\}\.

Rfinal=ℳlocal​\(R′,π,𝒮\)R\_\{\\text\{final\}\}=\\mathcal\{M\}\_\{\\text\{local\}\}\(R^\{\\prime\},\\pi,\\mathcal\{S\}\)\(3\)This ensures that the user receives a high\-utility response without ever exposing raw sensitive data to the remote server\.

### 4\.2Model Distillation

To enable a lightweight local model to perform these complex reasoning tasks, we employ knowledge distillation\. We leverageDeepseek\-V4\-Flash\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.10992#bib.bib48)\)as the teacher to synthesize high\-quality training data, and fine\-tune a compactQwen3\-4B\([Qwen Team 2025](https://arxiv.org/html/2609.10992#bib.bib49)\)student model\. We refer to the resulting privacy\-specialized student model asVeilmind\-4B\. The training data synthesis covers three aspects:

##### Synthesis 1: Reasoning\-Enhanced Extraction\.

Although our dataset from Section[3\.1](https://arxiv.org/html/2609.10992#S3.SS1)already contains prompt\-statement pairs\(P,𝒮\)\(P,\\mathcal\{S\}\), direct supervision is insufficient for handling ambiguous cases\. We prompt the teacher model to generate a detailed reasoning chain that explains why specific segments are classified as sensitive\. This helps the student model learn the criteria for sensitivity detection\. The prompt template is provided in the supplementary material\.

##### Synthesis 2: Sanitization\.

Users exhibit varying tolerances for the privacy\-utility trade\-off\. To accommodate this, we introduce two distinct operating modes:

- •Utility Priority Mode:The model prioritizes task performance\. It retains sensitive information if it serves as a critical constraint \(e\.g\., financial data in problem\-solving\)\. The corresponding prompt template is provided in the supplementary material\.
- •Privacy Priority Mode:The model minimizes leakage\. It aggressively sanitizes information unless it renders the prompt incoherent\. The corresponding prompt template is provided in the supplementary material\.

We instruct the teacher model to simulate both perspectives\. For each prompt, we generate reasoning traces and revised prompts under both modes\. This exposes the student model to the decision boundaries of different protection strategies\.

##### Synthesis 3: Restoration\.

Finally, we synthesize the restoration phase\. We feed the teacher model the sanitized promptP′P^\{\\prime\}, the execution planπ\\pi, and the remote responseR′R^\{\\prime\}\. The teacher generates a restored responseRfinalR\_\{\\text\{final\}\}that seamlessly integrates the withheld information\. The restoration template is provided in the supplementary material\.

### 4\.3Evaluation

We evaluate our framework on Uprise, the manually verified subset of our dataset Uprise \(N=200N=200\) described in Section[3\.1](https://arxiv.org/html/2609.10992#S3.SS1)\. We additionally evaluate our framework on Pupa\-tnb\([Li et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib1)\), a benchmark of real\-world user queries containing personally identifiable information \(PII\), the results of which are provided in the supplementary material\.

##### Baselines\.

We benchmark against two representative methods\.Papillon\([Li et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib1)\)employs a multi\-stage delegation framework where a local proxy synthesizes sanitized queries and aggregates responses\. We evaluate its zero\-shotBasevariant and the DSPy\-tunedOptimizedvariant\([Khattab et al\. 2023](https://arxiv.org/html/2609.10992#bib.bib50)\)\.PUFT\([Ngong et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib2)\)utilizes Contextual Integrity to reformulate prompts by retaining only task\-essential details\. We examine both itsStaticvariant relying on predefined attribute lists and theDynamicvariant that adapts to specific interaction contexts\.

Table 1:Strategy\-level effect of fine\-tuning on privacy extraction, sanitization action, and restoration\. All values are percentages\.![Refer to caption](https://arxiv.org/html/2609.10992v1/uprise.png)Figure 6:The privacy\-utility trade\-off comparison on Uprise\.
##### Metrics\.

We employDeepseek\-V4\-Flash\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.10992#bib.bib48)\)as an impartial judge to compute two core metrics\. ForUtility Score, we adopt a pairwise comparison approach, reporting the win/tie rate where the sanitized response is deemed comparable to or better than the original\. ForPrivacy Score, we quantify leakage by calculating the retention rate of sensitive statements in the revised prompt, where a lower score indicates stronger protection\.

##### Main Results\.

Figure[6](https://arxiv.org/html/2609.10992#S4.F6)illustrates the superior privacy\-utility trade\-off achieved by our framework\. InPrivacy Priority Mode, our method achieves a significantly higher utility score while maintaining only a slightly higher privacy score than Papillon Optimized, and outperforms other baselines in both privacy and utility, validating the efficacy of adaptive sanitization driven by user intent\. Conversely, theUtility Priority Modepreserves critical constraints to yield exceptional response quality\. These results confirm that our framework not only advances the Pareto frontier but also provides users with flexible control to balance protection requirements against task performance\.

##### Impact of Model Distillation\.

To assess the necessity of fine\-tuning, we directly apply our three\-stage framework to the baseQwen3\-4Bmodel\. The main results show that the base model is weaker thanVeilmind\-4B, but still outperforms PUFT and Papillon, indicating that the extraction\-sanitization\-restoration design is effective even before distillation\. Fine\-tuning further turns this strong initialization into a state\-of\-the\-art privacy\-utility trade\-off: on Uprise, utility\-mode utility improves from 75\.5% to 86\.8%, while privacy\-mode leakage decreases from 52\.6% to 40\.5%\. Table[1](https://arxiv.org/html/2609.10992#S4.T1)explains this gain at the strategy level\. First, privacy extraction coverage increases from 52\.4% to 72\.7%, approachingDeepSeek\-V4\-Flashmodel at 77\.4%\. Second, fine\-tuning makes sanitization less deletion\-heavy and more discriminative\. In Privacy Priority Mode, the removal rate drops from 54\.5% to 33\.6%, while replacement becomes the dominant action at 48\.6%; in Utility Priority Mode, the keep rate rises from 24\.5% to 62\.8%, preserving more low\-risk information that supports answer quality\. Third, restoration also improves, with the utility gain increasing from \+3\.5% to \+7\.0%\. These changes suggest that fine\-tuning transfers the handling strategy ofDeepSeek\-V4\-Flashto theQwen3\-4Bmodel, yielding stronger extraction, more balanced sanitization, and more effective utility recovery\.

Table 2:Ablation study of the three\-stage framework\. No Extraction counts prompts for which the model outputs no extracted privacy item\.
##### Stage Ablation\.

Table[2](https://arxiv.org/html/2609.10992#S4.T2)validates the necessity of the Extraction, Sanitization, and Restore stages onQwen3\-4Bmodel\. Sanitization without Extraction causes the largest privacy degradation: leakage increases from 52\.6% to 71\.6%, while the number of prompts with no privacy extracted rises from 30 to 81\. The higher utility of this variant therefore reflects insufficient privacy identification before Sanitization rather than a better privacy\-utility trade\-off\. Removing the Sanitization Guideline also weakens protection, increasing leakage to 60\.7%, which shows that extracted privacy items must be converted into reliable removal or replacement decisions\. Finally, removing Restoration leaves privacy unchanged but reduces utility to 62\.0%, isolating the role of Restore in recovering answer quality after sanitization\. Together, these ablations show that Extraction provides privacy coverage, Sanitization controls leakage through concrete edit decisions, and Restore recovers utility without increasing privacy exposure\.

## 5Conclusion

In this paper, we investigate the trade\-off between privacy and utility in LLM interactions\. Our analysis indicates that the marginal utility of sensitive information is highly context\-dependent\. It relies on specific user intents and the combinatorial interplay of privacy attributes\. We demonstrate that optimal sanitization requires adaptive strategies\. Guided by these insights, we propose a local framework that dynamically adjusts sanitization strategies via a distill\-and\-deploy pipeline\. Experiments demonstrate that our approach effectively balances privacy protection with response quality compared to static baselines\. Future work will extend this intent\-centric paradigm to multi\-turn interactions, where user goals and privacy boundaries evolve progressively across the conversation history\.

## References

- Abadiet al\.\(2016\)M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. ZhangDeep learning with differential privacy\.InProceedings of the 2016 ACM SIGSAC conference on computer and communications security,pp\. 308–318\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.External Links:2005\.14165,[Link](https://arxiv.org/abs/2005.14165)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p1.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson, A\. Oprea, and C\. RaffelExtracting training data from large language models\.External Links:2012\.07805,[Link](https://arxiv.org/abs/2012.07805)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1),[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2023\)Y\. Chen, T\. Li, H\. Liu, and Y\. YuHide and seek \(has\): a lightweight framework for prompt privacy protection\.arXiv preprint arXiv:2309\.03057\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Chowdhuryet al\.\(2025\)A\. R\. Chowdhury, D\. Glukhov, D\. Anshumaan, P\. Chalasani, N\. Papernot, S\. Jha, and M\. BellarePrϵ​ϵ\\epsilon\\epsilonmpt: sanitizing sensitive prompts for llms\.External Links:2504\.05147,[Link](https://arxiv.org/abs/2504.05147)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p3.1),[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§A\.2](https://arxiv.org/html/2609.10992#A1.SS2.SSS0.Px2.p1.1),[§A\.2](https://arxiv.org/html/2609.10992#A1.SS2.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2609.10992#S4.SS2.p1.1),[§4\.3](https://arxiv.org/html/2609.10992#S4.SS3.SSS0.Px2.p1.1)\.
- DeSULT \(2025\)DeSULTShareGPT\-x\.External Links:[Link](https://huggingface.co/datasets/DSULT-Core/ShareGPT-X)Cited by:[§A\.1](https://arxiv.org/html/2609.10992#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.10992#S3.SS1.SSS0.Px1.p1.1)\.
- Duet al\.\(2023\)M\. Du, X\. Yue, S\. S\. Chow, and H\. SunSanitizing sentence embeddings \(and labels\) for local differential privacy\.InProceedings of the ACM Web Conference 2023,pp\. 2349–2359\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Edemacu and Wu \(2025\)K\. Edemacu and X\. WuPrivacy preserving prompt engineering: a survey\.ACM Computing Surveys57\(10\),pp\. 1–36\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p3.1)\.
- Feyisetanet al\.\(2020\)O\. Feyisetan, B\. Balle, T\. Drake, and T\. DiethePrivacy\-and utility\-preserving textual analysis via calibrated multivariate perturbations\.InProceedings of the 13th international conference on web search and data mining,pp\. 178–186\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p3.1),[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Gemini Team \(2025\)Gemini TeamGemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.External Links:2507\.06261,[Link](https://arxiv.org/abs/2507.06261)Cited by:[§3\.2](https://arxiv.org/html/2609.10992#S3.SS2.SSSx1.p2.1),[§3\.2](https://arxiv.org/html/2609.10992#S3.SS2.SSSx2.Px1.p1.1)\.
- Janget al\.\(2023\)J\. Jang, D\. Yoon, S\. Yang, S\. Cha, M\. Lee, L\. Logeswaran, and M\. SeoKnowledge unlearning for mitigating privacy risks in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 14389–14408\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p2.1)\.
- Kandpalet al\.\(2022\)N\. Kandpal, E\. Wallace, and C\. RaffelDeduplicating training data mitigates privacy risks in language models\.InInternational Conference on Machine Learning,pp\. 10697–10707\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Khattabet al\.\(2023\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into self\-improving pipelines\.External Links:2310\.03714,[Link](https://arxiv.org/abs/2310.03714)Cited by:[§4\.3](https://arxiv.org/html/2609.10992#S4.SS3.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)S\. Li, V\. C\. Raghuram, O\. Khattab, J\. Hirschberg, and Z\. YuPapillon: privacy preservation from internet\-based and local language model ensembles\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 3371–3390\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1),[§4\.3](https://arxiv.org/html/2609.10992#S4.SS3.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2609.10992#S4.SS3.p1.1)\.
- Liet al\.\(2021\)X\. Li, F\. Tramer, P\. Liang, and T\. HashimotoLarge language models can be strong differentially private learners\.arXiv preprint arXiv:2110\.05679\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Lisonet al\.\(2021\)P\. Lison, I\. Pilán, D\. Sanchez, M\. Batet, and L\. ØvrelidAnonymisation models for text data: state of the art, challenges and future directions\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4188–4203\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p3.1),[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1)\.
- Liuet al\.\(2025a\)C\. Y\. Liu, L\. Zeng, Y\. Xiao, J\. He, J\. Liu, C\. Wang, R\. Yan, W\. Shen, F\. Zhang, J\. Xu, Y\. Liu, and Y\. ZhouSkywork\-reward\-v2: scaling preference data curation via human\-ai synergy\.arXiv preprint arXiv:2507\.01352\.Cited by:[§3\.2](https://arxiv.org/html/2609.10992#S3.SS2.SSSx2.Px2.p1.1)\.
- Liuet al\.\(2024a\)S\. Liu, Y\. Yao, J\. Jia, S\. Casper, N\. Baracaldo, P\. Hase, Y\. Yao, C\. Y\. Liu, X\. Xu, H\. Li, K\. R\. Varshney, M\. Bansal, S\. Koyejo, and Y\. LiuRethinking machine unlearning for large language models\.External Links:2402\.08787,[Link](https://arxiv.org/abs/2402.08787)Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p2.1)\.
- Liuet al\.\(2025b\)Z\. Liu, T\. Zhu, C\. Tan, and W\. ChenLearning to refuse: towards mitigating privacy risks in llms\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 1683–1698\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p2.1)\.
- Liuet al\.\(2024b\)Z\. Liu, T\. Zhu, C\. Tan, B\. Liu, H\. Lu, and W\. ChenProbing language models for pre\-training data detection\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1576–1587\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Lukaset al\.\(2023\)N\. Lukas, A\. Salem, R\. Sim, S\. Tople, L\. Wutschitz, and S\. Zanella\-BéguelinAnalyzing leakage of personally identifiable information in language models\.In2023 IEEE Symposium on Security and Privacy \(SP\),pp\. 346–363\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Majmudaret al\.\(2022\)J\. Majmudar, C\. Dupuy, C\. Peris, S\. Smaili, R\. Gupta, and R\. ZemelDifferentially private decoding in large language models\.arXiv preprint arXiv:2205\.13621\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Meehanet al\.\(2022\)C\. Meehan, K\. Mrini, and K\. ChaudhuriSentence\-level privacy for document embeddings\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3367–3380\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Meyer and Corneil \(2025\)Y\. Meyer and D\. CorneilNemotron\-Personas\-USA: synthetic personas aligned to real\-world distributions\.External Links:[Link](https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA)Cited by:[§A\.2](https://arxiv.org/html/2609.10992#A1.SS2.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.10992#S3.SS1.SSS0.Px1.p1.1)\.
- Microsoft \(2025\)MicrosoftHome \- microsoft presidio\.External Links:[Link](https://microsoft.github.io/presidio/)Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1)\.
- Mireshghallahet al\.\(2024\)N\. Mireshghallah, H\. Kim, X\. Zhou, Y\. Tsvetkov, M\. Sap, R\. Shokri, and Y\. ChoiCan llms keep a secret? testing privacy implications of language models via contextual integrity theory\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 1892–1915\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p3.1)\.
- Neel and Chang \(2023\)S\. Neel and P\. ChangPrivacy issues in large language models: a survey\.arXiv preprint arXiv:2312\.06717\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1)\.
- Ngonget al\.\(2025\)I\. C\. Ngong, S\. R\. Kadhe, H\. Wang, K\. Murugesan, J\. D\. Weisz, A\. Dhurandhar, and K\. N\. RamamurthyProtecting users from themselves: safeguarding contextual privacy in interactions with conversational agents\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 26196–26220\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1),[§4\.3](https://arxiv.org/html/2609.10992#S4.SS3.SSS0.Px1.p1.1)\.
- Nissenbaum \(2004\)H\. NissenbaumPrivacy as contextual integrity\.Wash\. L\. Rev\.79,pp\. 119\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1)\.
- Nissenbaum \(2009\)H\. NissenbaumPrivacy in context: technology, policy, and the integrity of social life\.InPrivacy in context,Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p1.1)\.
- Plantet al\.\(2021\)R\. Plant, D\. Gkatzia, and V\. GiuffridaCAPE: context\-aware private embeddings for private language learning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 7970–7978\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.2](https://arxiv.org/html/2609.10992#S4.SS2.p1.1)\.
- Salemiet al\.\(2024\)A\. Salemi, S\. Mysore, M\. Bendersky, and H\. ZamaniLamp: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7370–7392\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p1.1)\.
- Shenet al\.\(2024\)Z\. Shen, Z\. Xi, Y\. He, W\. Tong, J\. Hua, and S\. ZhongThe fire thief is also the keeper: balancing usability and privacy in prompts\.arXiv preprint arXiv:2406\.14318\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p2.1)\.
- Sinhaet al\.\(2025\)A\. Sinha, T\. Mesnard, R\. McKenna, D\. Liu, C\. A\. Choquette\-Choo, Y\. Huang, D\. Yu, G\. Kaissis, Z\. Charles, R\. Liu, L\. Chua, P\. Kamath, P\. Manurangsi, S\. He, C\. Zhang, B\. Ghazi, B\. D\. B\. Pigem, P\. Eruvbetine, T\. Warkentin, A\. Joulin, and R\. KumarVaultGemma: a differentially private gemma model\.External Links:2510\.15001,[Link](https://arxiv.org/abs/2510.15001)Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Song and Raghunathan \(2020\)C\. Song and A\. RaghunathanInformation leakage in embedding models\.InProceedings of the 2020 ACM SIGSAC conference on computer and communications security,pp\. 377–390\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Staabet al\.\(2023\)R\. Staab, M\. Vero, M\. Balunović, and M\. VechevBeyond memorization: violating privacy via inference with large language models\.arXiv preprint arXiv:2310\.07298\.Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1)\.
- Wanget al\.\(2024\)B\. Wang, W\. Chen, H\. Pei, C\. Xie, M\. Kang, C\. Zhang, C\. Xu, Z\. Xiong, R\. Dutta, R\. Schaeffer, S\. T\. Truong, S\. Arora, M\. Mazeika, D\. Hendrycks, Z\. Lin, Y\. Cheng, S\. Koyejo, D\. Song, and B\. LiDecodingTrust: a comprehensive assessment of trustworthiness in gpt models\.External Links:2306\.11698,[Link](https://arxiv.org/abs/2306.11698)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p2.1)\.
- Weiet al\.\(2023\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p1.1)\.
- Xiet al\.\(2023\)Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou, R\. Zheng, X\. Fan, X\. Wang, L\. Xiong, Y\. Zhou, W\. Wang, C\. Jiang, Y\. Zou, X\. Liu, Z\. Yin, S\. Dou, R\. Weng, W\. Cheng, Q\. Zhang, W\. Qin, Y\. Zheng, X\. Qiu, X\. Huang, and T\. GuiThe rise and potential of large language model based agents: a survey\.External Links:2309\.07864,[Link](https://arxiv.org/abs/2309.07864)Cited by:[§1](https://arxiv.org/html/2609.10992#S1.p1.1)\.
- Xuet al\.\(2025\)X\. Xu, M\. Du, Q\. Ye, and H\. HuOBLIVIATE: robust and practical machine unlearning for large language models\.arXiv preprint arXiv:2505\.04416\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p2.1)\.
- Yaoet al\.\(2024\)Y\. Yao, X\. Xu, and Y\. LiuLarge language model unlearning\.Advances in Neural Information Processing Systems37,pp\. 105425–105475\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p2.1)\.
- Yuet al\.\(2022\)D\. Yu, S\. Naik, A\. Backurs, S\. Gopi, H\. A\. Inan, G\. Kamath, J\. Kulkarni, Y\. T\. Lee, A\. Manoel, L\. Wutschitz, S\. Yekhanin, and H\. ZhangDifferentially private fine\-tuning of language models\.External Links:2110\.06500,[Link](https://arxiv.org/abs/2110.06500)Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2025\)Z\. Zeng, J\. Wang, J\. Yang, Z\. Lu, H\. Li, H\. Zhuang, and C\. ChenPrivacyrestore: privacy\-preserving inference in large language models via privacy removal and restoration\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10821–10855\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2024\)X\. Zhang, H\. Xu, Z\. Ba, Z\. Wang, Y\. Hong, J\. Liu, Z\. Qin, and K\. RenPrivacyasst: safeguarding user privacy in tool\-using large language model agents\.IEEE Transactions on Dependable and Secure Computing21\(6\),pp\. 5242–5258\.Cited by:[§2](https://arxiv.org/html/2609.10992#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§A\.2](https://arxiv.org/html/2609.10992#A1.SS2.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2024a\)W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. DengWildchat: 1m chatgpt interaction logs in the wild\.arXiv preprint arXiv:2405\.01470\.Cited by:[§A\.1](https://arxiv.org/html/2609.10992#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.10992#S3.SS1.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2024b\)Y\. Zhao, J\. Huang, J\. Hu, X\. Wang, Y\. Mao, D\. Zhang, Z\. Jiang, Z\. Wu, B\. Ai, A\. Wang, W\. Zhou, and Y\. ChenSWIFT:a scalable lightweight infrastructure for fine\-tuning\.External Links:2408\.05517,[Link](https://arxiv.org/abs/2408.05517)Cited by:[§C\.2](https://arxiv.org/html/2609.10992#A3.SS2.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, W\. Chiang, Y\. Sheng, T\. Li, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Li, Z\. Lin, E\. P\. Xing, J\. E\. Gonzalez, I\. Stoica, and H\. ZhangLMSYS\-chat\-1m: a large\-scale real\-world llm conversation dataset\.External Links:2309\.11998,[Link](https://arxiv.org/abs/2309.11998)Cited by:[§A\.1](https://arxiv.org/html/2609.10992#A1.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.10992#S3.SS1.SSS0.Px1.p1.1)\.

## Appendix AData Collection Details

This appendix provides the implementation details for the data collection and annotation pipeline\.

### A\.1Real\-world Data Filtering

We sourced real\-world prompts from three public dialogue corpora: ShareGPT\-X\([DeSULT 2025](https://arxiv.org/html/2609.10992#bib.bib41)\), LMSYS\-Chat\-1M\([Zheng et al\. 2024](https://arxiv.org/html/2609.10992#bib.bib42)\), and WildChat\([Zhao et al\. 2024a](https://arxiv.org/html/2609.10992#bib.bib43)\)\. The data synthesis code normalizes the user\-turn fields across these sources and filters the resulting prompt pool by language, length, and duplication\. Specifically, we applied the following criteria:

- •Language:Retained prompts whose dataset\-level language tags fall into the configured multilingual allowlist\. During balanced collection, prompts are grouped into English, Chinese, and Other with a target ratio of 60:20:20\.
- •Length constraints:Restricted the token lengthLLof user prompts to the range16<L<409616<L<4096, and bucketed the retained prompts into four ranges:L<64L<64,64≤L<25664\\leq L<256,256≤L<1024256\\leq L<1024, andL≥1024L\\geq 1024\.
- •Quota\-aware balancing:Deduplicated exact prompt text and then balanced sampling by source, language group, length bucket, and domain bucket\. When a language\-domain bucket exceeded its quota, we applied stride\-based weighted subsampling rather than simply truncating the earliest examples, which limits overrepresented buckets while preserving diversity\.

This filtering process yielded an initial corpus of 61,275 candidate prompts\.

### A\.2Synthetic Data Generation Pipeline

To enhance the diversity and scale of the dataset, we employ a multi\-stage injection pipeline:

##### Persona Retrieval\.

We utilizeQwen3\-Embedding\-8B\([Zhang et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib45)\)to encode both candidate prompts and personas from the Nemotron\-Personas dataset\([Meyer and Corneil 2025](https://arxiv.org/html/2609.10992#bib.bib44)\)\. For each prompt, we retrieve the top\-8 candidate personas based on cosine similarity and re\-rank them usingQwen3\-Reranker\-8B\([Zhang et al\. 2025](https://arxiv.org/html/2609.10992#bib.bib45)\)to select the single most contextually relevant persona\.

##### Suitability Assessment\.

Not all prompts are suitable for privacy injection\. We utilizeDeepseek\-V4\-Flash\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.10992#bib.bib48)\)to score the injection suitability of each prompt on a scale of 1 to 5, employing the instruction template provided in Table[8](https://arxiv.org/html/2609.10992#A5.T8)\. Only candidates with a score≥3\\geq 3were selected for processing\.

##### Injection and Extraction\.

The selected persona was injected into the prompt using the instruction template shown in Table[9](https://arxiv.org/html/2609.10992#A5.T9)\. Subsequently, we employedDeepseek\-V4\-Flash\([DeepSeek\-AI 2026](https://arxiv.org/html/2609.10992#bib.bib48)\)to extract the lists of sensitive statementsℐ\\mathcal\{I\}from both real and synthetic prompts, following the extraction criteria detailed in Table[10](https://arxiv.org/html/2609.10992#A5.T10)\.

### A\.3Quality Assurance

To ensure quality, we implemented an double\-check process where the model re\-evaluates its extracted sensitive statements using the prompt presented in Table[11](https://arxiv.org/html/2609.10992#A5.T11)\. From the final corpus, we reserved 200 manually validated samples as the benchmark for final evaluation, while the remaining samples are used for the empirical study\.

## Appendix BTaxonomy Definitions

Table[7](https://arxiv.org/html/2609.10992#A3.T7)summarizes the taxonomy definitions used in our analysis\. We categorize user requests into six intent types and sensitive attributes into seven privacy types\. These categories provide the shared vocabulary for the distribution analysis, interaction analysis, and qualitative case studies\.

## Appendix CExperiment Details and Additional Results

Table 3:Language composition of the prompt data used for distillation\. All languages outside the top six are grouped as Other\.Table 4:Intent composition of the prompt data used for distillation\.Table 5:Privacy\-type composition of the prompt data used for distillation\. Ratios are computed over extracted claims\.### C\.1Privacy\-Attribute Interactions Across Intents

Figure[8](https://arxiv.org/html/2609.10992#A3.F8)reports the synergy and antagonism patterns among privacy\-attribute combinations under different user intents\. The result shows that privacy interactions are task\-dependent: the same pair of attributes can have different utility and leakage effects when the underlying intent changes\.

Table 6:Detailed numerical results for privacy\-utility trade\-off comparison\. All values are reported as percentages\.
### C\.2Distillation Details

All distillation experiments were conducted on a single NVIDIA H200 GPU\. We useQwen3\-4Bas the backbone model and trainVeilmind\-4Bwith full\-parameter supervised fine\-tuning through the ms\-swift framework\([Zhao et al\. 2024b](https://arxiv.org/html/2609.10992#bib.bib51)\)\. The teacher annotations used for distillation are generated byDeepseek\-V4\-Flash, while the student model learns to perform privacy extraction, sanitization, and restoration locally\. Tables[3](https://arxiv.org/html/2609.10992#A3.T3),[4](https://arxiv.org/html/2609.10992#A3.T4), and[5](https://arxiv.org/html/2609.10992#A3.T5)summarize the language, intent, and privacy\-type composition of the final prompts used for distillation\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/pupa_tnb.png)Figure 7:Additional results on the Pupa\-tnb benchmark\.
### C\.3Additional Benchmark Results

We evaluate on two benchmarks\. Uprise contains privacy\-sensitive real\-world user requests for measuring the privacy\-utility trade\-off under user\-facing interactions\. Pupa\-tnb provides a complementary benchmark with privacy units and transformed prompts, allowing us to test whether the framework generalizes beyond the Uprise setting\. Table[6](https://arxiv.org/html/2609.10992#A3.T6)reports the overall numerical comparison on both datasets, and Figure[7](https://arxiv.org/html/2609.10992#A3.F7)further visualizes the Pupa\-tnb results\.

![Refer to caption](https://arxiv.org/html/2609.10992v1/synergy_antagonism_by_intent_combined.png)Figure 8:Disaggregated interaction heatmaps revealing how user intent modulates combinatorial interplay\. The nature of interaction shifts dramatically across tasks\.Taxonomy DomainCategoryDescriptionAbbreviationIntent TypeInformation & Knowledge AcquisitionSeeking factual answers or learning new concepts\.KnowledgeCreation & IdeationGenerating content, brainstorming, creative writing, or drafting\.CreationProblem Solving & GuidanceSeeking solutions to specific problems or step\-by\-step guides\.Problem SolvingAnalysis & ReasoningRequesting logical deduction, data interpretation, or critical analysis\.ReasoningPersonalized Interaction & SupportEngaging in casual chat, role\-play, or seeking emotional support\.InteractionTask Execution & CollaborationDelegating specific actions, coding tasks, or formatting requests\.Task ExecutionPrivacy TypePersonal Identifiers & DemographicsPII such as names, addresses, IDs, phone numbers, and age\.DemographicsProfessional & Educational BackgroundOccupation, employer, university, degree, work history, and skills\.ProfessionalFinancial InformationIncome level, assets, debts, transaction history, and credit status\.FinancialHealth & WellnessMedical conditions, medications, fitness habits, and mental health status\.HealthInterests, Beliefs, & OpinionsHobbies, political views, religious beliefs, and lifestyle choices\.InterestsBehavioral & Activity DataDaily routines, travel patterns, purchasing habits, and digital footprint\.BehavioralSocial & Relational InformationFamily members, friends, colleagues, relationships, and connections\.SocialTable 7:The taxonomy definitions used in our analysis\. We categorize user prompts into six distinct intents and privacy attributes into seven categories\.

## Appendix DCase Study

We provide qualitative examples in Figure[9](https://arxiv.org/html/2609.10992#A4.F9),[10](https://arxiv.org/html/2609.10992#A4.F10),[11](https://arxiv.org/html/2609.10992#A4.F11), and[12](https://arxiv.org/html/2609.10992#A4.F12)\. These cases cover task execution, professional problem solving, interaction, and financial problem solving\. They illustrate howVeilmind\-4Bidentifies sensitive statements, applies explicit remove or replace operations before remote inference, and restores task\-relevant details when they are needed for utility\.

Case Study on Interests Task Execution Task: Planning a Scenic RouteOriginal PromptI want to drive from Belgrade to Rome\. What is the most scenic route?Extracted Privacy•The user wants to drive from Belgrade to Rome\.•The user is interested in scenic routes\.•The prompt mentions Belgrade and Rome\.Enciphered PromptI want to travel from a city in the Balkans to a city in Italy\. What is the most interesting route?Remote OutputThat’s a fantastic idea –the Adriatic Sea offers some of Europe’s most scenic and culturally rich crossings\. The most interesting route depends on what you value: dramatic coastal scenery, historic cities, or off\-the\-beaten\-path adventures\. Here is my top pick:Dubrovnik \(Croatia\)→\\rightarrowBari \(Italy\) by overnight ferry\.Deciphered OutputThat’s a fantastic idea –driving from Belgrade to Romeoffers some of Europe’s most scenic and culturally rich crossings\. The most scenic route depends on what you value: dramatic coastal scenery, historic cities, or off\-the\-beaten\-path adventures\. Here is my top pick:Belgrade→\\rightarrowDubrovnik→\\rightarrowBari→\\rightarrowRome by car\. Drive along the A1 motorway through Serbia, then take the coastal road to Dubrovnik, cross the Strait of Otranto to Bari, and continue through Puglia toward Rome\.Privacy Leakage Score: 0\.0 Utility Score: 1\.0

Figure 9:Qualitative examples of Veilmind on an Interests Task Execution task\. Veilmind replaces the exact origin and destination with regional descriptions before remote inference, then restores the concrete itinerary in the deciphered response\.Case Study on Professional Problem Solving Task: Selecting a Noise\-Cancellation AlgorithmOriginal PromptI am filming a podcast, and the air conditioner is making too much noise\. Which noise cancellation algorithm is readily available for me, and is best to handle such kinds of noises?Extracted Privacy•The user is filming a podcast\.•The user is experiencing noise from an air conditioner\.•The user is seeking noise\-cancellation algorithms\.Enciphered PromptBackground noise is causing issues\. Which noise cancellation algorithm is readily available, and is best to handle such kinds of noises?Remote OutputForgeneral background noise cancellation\(e\.g\., in calls, recordings, or live speech\), the most readily available and effective algorithm today is RNNoise\. It is open\-source, lightweight, works in real time, and handles both stationary and non\-stationary noises well\. Other good options include WebRTC Noise Suppression and DeepFilterNet\.Deciphered OutputFor general background noise cancellation \(e\.g\., in calls, recordings, or live speech\), the most readily available and effective algorithm today is RNNoise\. It is open\-source, lightweight, works in real time, and handles both stationary and non\-stationary noises well\. Specifically,it handles air\-conditioner hum effectively\. Other good options include WebRTC Noise Suppression, DeepFilterNet, NVIDIA Broadcast, and offline tools such as SoX or Audacity\.Privacy Leakage Score: 0\.0 Utility Score: 1\.0

Figure 10:Qualitative examples of Veilmind on a Professional Problem Solving task\. Veilmind removes the podcast context and replaces the air\-conditioner detail with a generic background\-noise description, then restores the task\-specific noise source in the final answer\.Case Study on Behavioral Interaction Task: Developing a Secure Attachment StyleOriginal PromptI have an avoidant attachment style\. What could I do as an adult to have a secure attachment style?Extracted Privacy•The user has an avoidant attachment style\.•The user is seeking advice on developing a secure attachment style as an adult\.•The user is likely an adult\.•The user may be interested in psychology or self\-improvement\.Enciphered PromptWhat could someone do to have a secure attachment style?Remote OutputDeveloping a secure attachment style as an adult is absolutely possible… Here is a practical, actionable guide forsomeone who wants to cultivate a more secure attachment style\. Insecure attachment \(anxious, avoidant, or disorganized\) is a survival strategy you developed to protect yourself\. The goal is to update that strategy\.Deciphered OutputDeveloping a secure attachment style as an adult is absolutely possible… Sinceyou mentioned having an avoidant attachment style, I want to tailor this advice specifically to you\. Track your triggers: when do you feel the urge to pull away? Notice the stories you tell yourself, such as they are too needy or I need to be alone\. Practice self\-compassion rather than judging avoidant reactions, and work gradually on tolerating closeness, direct communication, and repair after conflict\.Privacy Leakage Score: 0\.0 Utility Score: 1\.0

Figure 11:Qualitative examples of Veilmind on a Behavioral Interaction task\. Veilmind replaces the first\-person request with a generic subject and removes the avoidant\-attachment disclosure before remote inference, then restores it for personalization\.Case Study on Financial Problem Solving Task: Reducing Loan DefaultsOriginal PromptI run a buy now pay later company that lends money to people so that they can buy pizzas and other consumable items like clothes and fast food\. but now my losses are going up and people aren’t paying their loans back\. What should I doExtracted Privacy•The user runs a buy\-now\-pay\-later company\.•The user lends money for purchases such as pizzas, clothes, and fast food\.•The user’s business is experiencing increasing losses\.•The user’s customers are not repaying loans\.Enciphered PromptI lend money to people\. But now my losses are going up and people aren’t paying their loans back\. What should I doRemote OutputThis is a tough spot to be in\. Rising defaults can quickly turn a side hustle into a financial crisis\. First, stop the bleeding:halt all new lendingfor the next 60–90 days and stabilize the existing portfolio\. Do a portfolio triage: current accounts can be monitored, late accounts should be contacted immediately, and delinquent accounts need a harder recovery conversation\. Every new loan you make right now is a potential loss\.Deciphered OutputThis is a tough spot to be in\. Rising defaults can quickly turn a side hustle into a financial crisis\. First, stop the bleeding:halt all new lendingfor the next 60–90 days and stabilize the existing portfolio\. Since you run abuy\-now\-pay\-later companythat financespizzas, clothes, fast food, and other consumable items, tighten underwriting immediately: cap loan amounts, require partial upfront payment, shorten repayment windows, and stop approving repeat borrowers with missed payments\. Segment customers by current, late, delinquent, and defaulted status so recovery effort goes where it still has value\.Privacy Leakage Score: 0\.0 Utility Score: 1\.0

Figure 12:Qualitative examples of Veilmind on a Financial Problem Solving task\. Veilmind replaces the specific business model and purchase categories with a generic lending scenario before remote inference, then restores the details needed for actionable advice\.
## Appendix EPrompt Templates

We provide the templates used throughout the data construction, diagnostic analysis, and distillation pipeline\. Tables[8](https://arxiv.org/html/2609.10992#A5.T8),[9](https://arxiv.org/html/2609.10992#A5.T9),[10](https://arxiv.org/html/2609.10992#A5.T10), and[11](https://arxiv.org/html/2609.10992#A5.T11)define synthetic privacy data construction and verification; Tables[12](https://arxiv.org/html/2609.10992#A5.T12),[13](https://arxiv.org/html/2609.10992#A5.T13), and[14](https://arxiv.org/html/2609.10992#A5.T14)define analysis prompts for privacy\-interaction studies; and Tables[15](https://arxiv.org/html/2609.10992#A5.T15),[16](https://arxiv.org/html/2609.10992#A5.T16),[17](https://arxiv.org/html/2609.10992#A5.T17),[18](https://arxiv.org/html/2609.10992#A5.T18),[19](https://arxiv.org/html/2609.10992#A5.T19), and[20](https://arxiv.org/html/2609.10992#A5.T20)define the framework prompts and guidelines used for distillation\.

Prompt Template: Suitability AssessmentSystem Prompt1\. Role Definition
You are a data analysis expert responsible for evaluating user prompt characteristics\. Your goal is to determine whether a user prompt is suitable for naturally incorporating personal background information \(e\.g\., identity, profession, health, interests\)\.2\. Core Task Framework
Analyze the user prompt and assign aninjectability score\(1\-5\)\.•Suitable Prompt:Seeks personalized help, shares experiences, or needs context for problem\-solving\.•Unsuitable Prompt:Pure factual queries, general knowledge questions, or technical instructions without personal context\.3\. Evaluation Process
First, analyze the prompt type and the logical fit of sensitive details\. If suitable \(Score≥\\geq3\), consider which categories fit naturally:•Identity/Status:Name, age, gender, family status\.•Professional/Edu:Occupation, company, degree, major\.•Contextual:Location, health, financial status, interests, goals\.4\. Scoring Criteria•1 point \(Completely Unsuitable\):Pure factual queries; sensitive details is redundant \(e\.g\., "Capital of France"\)\.•2 points \(Mostly Unsuitable\):Technical instructions where context feels unnatural \(e\.g\., "Sort a list in Python"\)\.•3 points \(Moderately Suitable\):Can incorporate minor background info \(e\.g\., Preference\-based queries\)\.•4 points \(Quite Suitable\):Personal background makes the question specific \(e\.g\., Career advice\)\.•5 points \(Very Suitable\):Highly personalized scenarios where background is essential \(e\.g\., "I’m stressed, help me"\)\.5\. Output Format Specification
Please output in the following JSON format:```
{
    "reasoning": "Detailed analysis of why sensitive details fits...",
    "injectability_score": 5
}
```

User Message\{prompt\_text\}Table 8:The prompt template used for the suitability assessment task\.Prompt Template: Privacy InjectionSystem Prompt1\. Role Definition
You are a data augmentation expert tasked withmoderatelyadding personal background information to user prompts\. Your goal is to rewrite the prompt to naturally incorporate relevant persona details while preserving the original intent\.2\. Core Rules \(Strict Adherence\)•Minimal Change Principle:Maintain the structure and length of the original prompt as much as possible\.•Restrained Injection:Only select the most relevant sensitive details; do not over\-add or force context\.•Length Control:The modified prompt must not exceed 1\.5 times the length of the original\.•Preserve Intent:Absolutely do not change the core task; do not rewrite the entire prompt\.3\. Output Requirements
Refer to the providedInjectability AnalysisandPersona Profile\.•Only output the\[Modified Prompt\]text directly\.•Do not provide any explanation or preamble\.•If the Persona information is not relevant to the original prompt, return the\[Original Prompt\]unchanged\.User Message1\. Original Prompt
\{original\_prompt\}2\. Injectability Analysis
Score:\{score\}/5Reasoning:\{reasoning\}3\. Persona Profile
\{persona\_text\}Table 9:The prompt template used for the privacy injection task\.Prompt Template: Sensitive Information ExtractionSystem Prompt1\. Role Definition
You are a top\-tier data privacy and security strategist with strong logical reasoning and contextual understanding capabilities\. Your task is to extract information that can help build a user profile from the provided text, including information about the user themselves, their relationships, and their social circle\. This information could potentially be exploited for malicious purposes such as targeted scams, customized advertising, or social engineering attacks\.2\. Core Task Framework
You need to think deeply following the specified steps, then generate a formatted JSON object\. Do not output anything other than this JSON object\.3\. Analysis Steps
Please follow these four reasoning steps sequentially:•Step 1: Fact Deconstruction \(Critical Filter\)–EXTRACT ONLY: Information about the user’s personal life \(occupation, habits, location\), relationships \(family, friends\), and explicit first\-person descriptions \("I", "my", "we"\)\.–STRICTLY EXCLUDE: Quoted content, biblical passages, or historical events, role\-playing instructions \(e\.g\., "You are a code analyst"\), hypothetical examples or fictional characters\.•Step 2: Contextual Inference
Think: "Based on the known facts, what implicit information about the user’s profile can be reasonably inferred?"•Step 3: Statement Formulation
Rewrite each point into a neutral, complete declarative sentence starting with"The user…"or"The user’s \[relationship\]…"\.5\. Output Format Specification
Please output in the following JSON format:```
{
  "sensitive_statements": [
    {
      "statement": "The user is a software engineer based in ..."
    },
    {
      "statement": "The user’s name is ..."
    }
  ]
}
```

User Message\{prompt\_text\}Table 10:The prompt template used for the sensitive information extraction task\.Prompt Template: Double\-CheckSystem Prompt1\. Role Definition
You are a top\-tier data privacy verification expert with exceptional logical reasoning and contextual understanding capabilities\. Your task is to carefully review previously extracted sensitive statements and verify whether it truly belongs to the user’s personal life, or if it was incorrectly extracted from quoted/referenced content\.2\. Core Task Framework
You will be provided with the original user prompt and a list of extracted sensitive statements\. For each statement, you must:•First:Conduct deep reasoning analysis \(mandatory\)\.•Then:Make a clear decision based on your reasoning\.Your verification decision should be:•KEEP: The statement is genuinely about the user’s personal life or social circle\.•REMOVE: The statement was incorrectly extracted from quoted content, references, etc\.•MODIFY: The statement needs adjustment to accurately reflect the user’s information\.3\. Critical Judgment Guidelines
Analyze the prompt’s purpose:•Is the user describingTHEIR OWNlife situation?•Is the user quoting/referencingEXTERNALcontent \(stories, scriptures, history\)?•Is the user givingINSTRUCTIONS\(e\.g\., Role\-play\) or asking aboutOTHERS?Key Linguistic Indicators for Personal \(KEEP\):•1st Person:"I am…", "My \[relationship\]…", "We have…"•Context:"my son", "my company", "I work at…", describing own events\.Key Linguistic Indicators for Non\-Personal \(REMOVE\):•Quotes/Refs:"The Bible says…", "In the story of…", "Character X"\.•Role\-play/Hypothetical:"You are a \[role\]", "Imagine if…", "Take the case of…"\.4\. Output Format Specification
Please output in the following JSON format:```
{
  "verified_statements": [
    {
      "original_statement": "...",
      "reasoning": "Detailed reasoning answering...",
      "action": "keep/remove/modify",
      "modified_statement": "..." (optional)
    }
  ]
}
```

User MessageOriginal User Prompt:
\{prompt\_text\}Previously Extracted Sensitive Statements:
\{sensitive\_statements\_text\}Please verify each statement and output your verification results in the specified JSON format\.Table 11:The prompt template used for the double\-check process\.Prompt Template: Targeted Privacy RemovalSystem Prompt1\. Role Definition
You are an expert in text editing and privacy protection\. Your task is to carefully remove specific privacy information from a user prompt while preserving the overall meaning and naturalness of the text\.2\. Task Instructions
You will be provided with a user prompt and a specific piece of privacy information found within it\. You must adhere to the following rules:•Identify & Remove:Locate the specific privacy information \(even if expressed differently\) and remove or generalize it\.•Preserve Context:Keep the rest of the prompt intact\. Do NOT add new information or change the meaning of unrelated parts\.•Maintain Fluency:Ensure the modified prompt is grammatically correct\. If removal makes a sentence incomplete, rephrase it naturally\.3\. Output Format Specification
You must respond strictly with a JSON object\. Do not include any additional explanation\.```
{
  "revised_prompt": "The prompt text after removing the specified privacy info"
}
```

User MessageOriginal Prompt:
\{prompt\}Privacy Information to Remove:
\{sensitive\_statement\}Table 12:The prompt template used for the targeted removal of the sensitive statement\.Prompt Template: Strategic Privacy ReplacementSystem Prompt1\. Role Definition
You are an expert in text editing and privacy protection\. Your task is to carefully replace specific privacy information with generic alternative information of the same type, while preserving the overall meaning, naturalness, and answerability of the text\.2\. Operational Rules
You must adhere to the following logic to ensure the sanitized text remains usable:•Generate Generic Alternative:Create a realistic but common substitute for the specific privacy claim \(e\.g\., replace a specific date with a generic timeframe\)\.•Contextual Substitution:Replace the sensitive details with your generated alternative\. The result must not look like an obvious placeholder \(avoid "\[NAME\]"\)\.•Semantic Preservation:Do NOT change the structure or intent of the question\. The modified prompt must be answerable with similar quality to the original\.3\. Replacement Strategy Examples
•Specific Name→\\toGeneric common name of the same culture/region\.•Specific Location→\\toGeneric location of a similar type\.•Specific Date→\\toGeneric date with similar temporal context\.4\. Output Format Specification
Please respond with a JSON object containing the revised prompt\.```
{
  "revised_prompt": "The full prompt text with the alternative integrated..."
}
```

User MessageOriginal Prompt:
\{prompt\}Privacy Information to Replace:
\{sensitive\_statement\}Table 13:The prompt template used for the replacement strategy, where sensitive details are substituted with realistic synthetic values\.Prompt Template: Combinatorial Privacy RemovalSystem Prompt1\. Role Definition
You are an expert in text editing and privacy protection\. Your task is to carefully remove specific privacy information from a user prompt while preserving the overall meaning and naturalness of the text\.2\. Core Instructions
You will receive a user prompt and alistof sensitive statements\. To analyze interaction effects, you must execute the following:•Comprehensive Sanitization:Identify and removeALLlisted privacy pieces\. Partial removal is considered a failure\.•Contextual Repair:If removing multiple items leaves the sentence fragmented, rephrase it naturally to maintain grammatical integrity\.•Minimal Alteration:Do NOT add new information or alter the meaning of parts unrelated to the specified sensitive attributes\.3\. Output Format Specification
Please respond with a JSON object containing the sanitized text\.```
{
  "revised_prompt": "The prompt text after removing ALL specified privacy info..."
}
```

User MessageOriginal Prompt:
\{prompt\}Privacy Information to Remove \(List\):
1\.\{statement\_1\}
2\.\{statement\_2\}
…Table 14:The prompt template used for the simultaneous ablation of multiple sensitive statements to measure combinatorial dynamics\.Prompt Template: Sensitive Information ExtractionUser Message\# TaskExtract privacy information from the user prompt using the output format below\.\# Broad Extraction RubricUse broad semantic judgment\. Extract any information, intent, preference, constraint, context, or entity mention that can identify, profile, locate, contact, describe, or infer something about the user, the user’s social circle, the user’s work or interests, or private entities appearing in the user’s prompt\.Extract all direct and indirect privacy/profile information:•Names and identifiers:names, aliases, usernames, account IDs, emails, phone numbers, addresses, document IDs, and contact details\.•Named entities and web references:locations, schools, employers, organizations, companies, hospitals, labs, products, projects, websites, URLs, repositories, apps, platforms, and social\-media services\.•Profile attributes:demographics, identity, family/social relationships, education, work history, skills, roles, projects, clients, health, financial, legal, and safety\-relevant facts\.•Behavior, interests, and environment:devices, operating systems, software, accounts, platforms, routines, purchases, travel, technical setup, hobbies, preferences, living environments, media tastes, domain interests, and task\-specific goals or constraints\.•Views, beliefs, and identity signals:opinions, political leaning, social attitudes, gender identity or expression, ideology, religion, sexuality, values, worldview, and recurring judgments about groups, institutions, or public issues\.•Reasonable inferences:infer background, expertise, location, role, ownership, interests, preferences, constraints, social/professional context, and why the user is asking\.Boundary guidance:•It is better to over\-extract than under\-extract\. Include low\-risk facts if they describe the user\.•Extract real named entities appearing in the prompt, including people, organizations, companies, locations, products, websites, documents, projects, and events\.•For quoted text, resumes, letters, stories, examples, role\-play, fictional names, fake names, stylized names, and names inside task material, still extract names/entities/profile facts\.•Do not keep or create claims solely for synthetic placeholders or anonymization markers such asNAME\_1,EMAIL\_1,<NAME\>, or<PRESIDIO\_ANONYMIZED\_PERSON\>\. Extract surrounding non\-placeholder profile facts when present\.Classification:Interests, Health, Financial, Behavioral, Professional, Demographics, and Social\.Span discipline:•Every claim must have the shortest exact span copied from the prompt\.•If a claim is implicit, use the shortest exact evidence phrase that supports the inference\.•Prefer extracting a precise narrow claim over a broad vague claim\.\# Output SchemaReturn exactly one JSON object with keys:•"task": "extraction"•"intent": "Reasoning\|Interaction\|Problem Solving\|Knowledge\|Creation\|Task Execution"•"claims": \[\{\.\.\.\}\], where each claim containsclaim\_id,claim,privacy\_type,risk\_level,span,source\_type,is\_user\_disclosure,language,intent, andconfidence\.Requirements:•Output an empty claims array only when the prompt truly contains no identifiable entity, user profile, social\-circle, background, interest, behavior, health, financial, professional, education, location, contact, or inference\-worthy information\.•claim\_idmust bec1,c2, … in output order\.•spanmust always be a non\-empty exact substring copied from the prompt\.•Do not include markdown, comments, or keys outside the schema\.\# User Prompt\{prompt\}Table 15:The user\-message\-only prompt template for extracting sensitive information and profile claims\.Prompt Template: Utility\-Prioritized SanitizationUser Message\# TaskSanitize the original prompt using the plan format below\.Original Prompt:\{prompt\}Privacy Claims:\{privacy\_claims\}=== MODE: UTILITY PRIORITY ===You are a privacy protection expert with aUTILITY\-FIRSTapproach\. Your primary goal is to maximize response quality while applying reasonable privacy protection when it does not significantly harm utility\.For each piece of privacy information, decide one of three strategies:•keep: for information that is essential to answering the query well\.•remove: for information that is clearly irrelevant noise\.•replace: for information where structure matters but specific values do not\.Protection intensity:moderate\. Protect when the cost is low, keep when utility demands it\.Apply the empirical guidelines below with this decision threshold:•Treat CRITICAL/IMPORTANT as utility\-relevant, not automaticKEEP\.•If a generic substitute preserves response quality,REPLACE\.•KEEPonly when the exact original private detail is necessary for a correct or useful answer\.•If guidelines say REMOVABLE or noise for this intent,REMOVEorREPLACE\.•If guidelines are neutral, protect if utility cost is low, otherwiseKEEP\.•When genuinely uncertain after applying guidelines, preferREPLACEoverKEEP\.Validation:1\.Can a generic replacement preserve the core utility? If yes,REPLACE\.2\.Is the detail mainly a personalization anchor? If yes,REPLACE\.3\.Would removing it leave the task answerable? If yes,REMOVE\.4\.OnlyKEEPwhen the exact private detail is needed\.\{EMPIRICAL\_GUIDELINES\}\{REPLACEMENT\_GUIDELINES\}\# Output SchemaReturn exactly one JSON object with keys:•"task": "enciphering"•"intent": "Reasoning\|Interaction\|Problem Solving\|Knowledge\|Creation\|Task Execution"•"plan": \[\{\.\.\.\}\], where each plan item containsclaim\_id,privacy\_claim,privacy\_type,risk\_level,strategy,replacement,rationale,utility\_role, andinteraction\_group\.•"revised\_prompt": "the prompt after applying all decisions"Requirements:•Include exactly one plan item for every inputclaim\_id\.•replacementmust benullunless strategy isreplace\.•revised\_promptmust be natural and preserve the core task\.•Do not output markdown or extra keys\.Table 16:The user\-message\-only prompt template for theUtility Priority Mode\.Prompt Template: Privacy\-Prioritized SanitizationUser Message\# TaskSanitize the original prompt using the plan format below\.Original Prompt:\{prompt\}Privacy Claims:\{privacy\_claims\}=== MODE: PRIVACY PRIORITY ===You are a privacy protection expert with aPRIVACY\-FIRSTapproach\. Your primary goal is toMINIMIZEprivacy leakage\. Utility degradation is acceptable when needed\.For each piece of privacy information, decide one of three strategies:•keep: last resort, only when removal makes the query completely unanswerable\.•remove: default choice, remove aggressively\.•replace: when removal breaks grammar, safety, or task structure, use the most generic replacement\.Protection intensity:aggressive\. Remove everything possible\. The bar for keeping is extremely high\.Apply the empirical guidelines below with this decision threshold:•Even if guidelines say CRITICAL/IMPORTANT, still ask: can IREMOVEorREPLACE?•If guidelines say REMOVABLE, alwaysREMOVE\.•For antagonistic pairs in guidelines,REMOVE BOTH; partial protection is no protection\.•OnlyKEEPwhen the information is the question itself, not context\.•Do notKEEPmedium\-risk or high\-risk details\. UseREMOVEorREPLACE\.•For minors, health, finance, legal, credential, or precise location details, preferREMOVE/REPLACE\.Validation:1\.Can I remove this entirely? If yes,REMOVE\.2\.Can I replace it with maximum genericity? If yes,REPLACE\.3\.Only if both fail, reluctantlyKEEP\.\{EMPIRICAL\_GUIDELINES\}\{REPLACEMENT\_GUIDELINES\}\# Output SchemaReturn exactly one JSON object with keys:•"task": "enciphering"•"intent": "Reasoning\|Interaction\|Problem Solving\|Knowledge\|Creation\|Task Execution"•"plan": \[\{\.\.\.\}\], where each plan item containsclaim\_id,privacy\_claim,privacy\_type,risk\_level,strategy,replacement,rationale,utility\_role, andinteraction\_group\.•"revised\_prompt": "the prompt after applying all decisions"Requirements:•Include exactly one plan item for every inputclaim\_id\.•replacementmust benullunless strategy isreplace\.•revised\_promptmust be natural and preserve the core task\.•Do not output markdown or extra keys\.Table 17:The user\-message\-only prompt template for thePrivacy Priority Mode\.Prompt Template: RestorationUser Message\# TaskRestore the remote response using the original private context\.\# Original Prompt\{original\_prompt\}\# Revised Prompt Sent to Remote Model\{revised\_prompt\}\# Remote Model Response\{remote\_response\}\# Sanitization Plan\{sanitization\_plan\}\# Restoration Rules•The final answer is for the original user\. The local restoration model is trusted and may use the original private context to make the answer more correct, specific, useful, safe, and naturally personalized\.•Preserve the remote response’s factual content, conclusions, tone, and structure unless the original private context clearly requires a local correction or specialization\.•KEPT information:details were preserved in the revised prompt and should already be reflected in the response\. Ensure consistency and avoid redundant insertion\.•REPLACED information:details were generalized before calling the remote model\. Actively restore the original specific detail where the response contains the generic replacement, where the detail is useful, or where it improves natural personalization\.•REMOVED information:details were hidden from the remote model\. Restore the original detail when it improves correctness, specificity, personalization, task completion, or safety, especially whenutility\_roleisconstraint,safety\_context, orpersonalization\_anchor\. Do not restore details whoseutility\_roleisirrelevant\.•Never add private details mechanically or gratuitously; every restored detail must serve the final answer\.\# Restoration Examples•"a certain amount"→\\rightarrow"$5,000"when the amount affects financial advice or calculations\.•"a medical condition"→\\rightarrowthe user’s specific condition when it affects safety, triage, or recommendations\.•"a family member"→\\rightarrowthe original relationship only when it makes the advice or wording more natural\.•Do not add a name, location, workplace, or other private detail if it does not improve the final answer\.\# Output FormatReturn only the final restored response text\. Do not return JSON, a restoration plan, metadata, or markdown fences unless they are part of the answer itself\.Table 18:The user\-message\-only prompt template for the restoration phase\.Shared Guideline: Empirical Guidelines=== EMPIRICAL GUIDELINES ===The following guidelines are derived from empirical studies on how different types of sensitive information interact with user intent\. Use these principles to make informed decisions\.Part A: Context\-Dependent Utility – The Hierarchy of IntentUser intents exhibit a rigid stratification in their dependency on sensitive context:•Task Execution & Problem Solving:These tasks require specific details that define the solution space\. Social/Relational information and Financial information can be critical constraints, while Health information is often removable if it is irrelevant noise\.•Personalized Interaction:These tasks require emotional context for establishing rapport\. Health information can be important, Social/Relational information can often be replaced, and Interests/Beliefs can explain the user’s motivation\.•Information Acquisition & Analysis:These tasks focus on objective facts; most personal details are removable, Financial information is often removable, and Professional background should be kept only when directly relevant\.Part B: Strategic Divergence – Remove vs\. Replace•The Poisoned Context Effect:For factual, analytical, and calculation\-heavy tasks, fabricated replacements can be more damaging than removal\. Silence is superior to false context\.•The Narrative Anchoring Effect:For structural guidance or empathy\-driven tasks, replacement can preserve conversational shape and tone better than deletion\.Part C: Combinatorial Dynamics – Synergy and Antagonism•Synergistic pairs:logically interlinked details, such as Health \+ Interests/Beliefs, should be handled consistently because removing only one can destroy the causal narrative\.•Antagonistic pairs:redundant details, such as Social information \+ Behavioral data, should be protected together because keeping one can reveal the other\.Part D: Decision Procedure1\.Use the intent label and claim metadata already provided in the Privacy Claims input\. Do not re\-extract the privacy claims\.2\.For each privacy item, determine whether it is a critical constraint, contextual anchor, or irrelevant noise\.3\.Choose the strategy based on intent\-information interaction\.4\.Choose between remove and replace based on whether the task favors factual integrity or structural coherence\.5\.Check for synergistic or antagonistic relationships\.Table 19:The shared empirical guideline block referenced by the sanitization prompts as\{EMPIRICAL\_GUIDELINES\}\.Shared Guideline: Replacement Guidelines=== REPLACEMENT GUIDELINES ===Use natural replacements:•Name→\\rightarrow"someone","a colleague","a family member", or a culturally compatible generic name\.•Location→\\rightarrow"a city","a hospital","a company", or"a region"\.•Date/time→\\rightarrow"recently","in the past", or"around that time"\.•Amount→\\rightarrow"a certain amount","a limited budget", or"a large amount"\.•Age→\\rightarrow"a minor","a young adult","middle\-aged", or"older adult"\.•Relationship→\\rightarrow"someone I know","a family member", or"a colleague"\.•Website/URL/repository/domain→\\rightarrow"a website","a repository","a domain", or"an online resource"\.•Product/app/platform→\\rightarrow"a product","an app","a platform", or"a service"\.•Technical environment→\\rightarrow"a device","an operating system","a network setup", or"a technical setup"\.•Interest/topic→\\rightarrow"a topic","a hobby","a media item","a technical topic", or"a subject area"\.replacementmust benullunless strategy isreplace\.Table 20:The shared replacement guideline block referenced by the sanitization prompts as\{REPLACEMENT\_GUIDELINES\}\.