PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Hugging Face Daily Papers Papers
privacy llm-agents alignment dataset human-annotation contextual-privacy

Summary

PrivacyAlign introduces a human-annotated dataset and training framework for aligning LLM agents to respect contextual privacy norms, showing that frontier models still leak sensitive information and that human-grounded evaluation improves alignment.

AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.
Original Article
View Cached Full Text

Cached at: 06/25/26, 05:13 PM

Paper page - PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Source: https://huggingface.co/papers/2606.21710 Agents are powerful because they can use tools to gather information, remember context, and act on our behalf. But those same capabilities create new privacy risks: an agent with access to emails, calendars, files, databases, tools, and memory may disclose sensitive information that should not have been shared.

I’m excited to share PrivacyAlign, our new work on privacy alignment for LLM agents.

Privacy is not simply the absence of disclosure. It depends on context: what information should flow, to whom, in what situation, and at what level of detail.

That makes privacy alignment especially challenging for LLMs, because the right answer often depends on scenario-specific social norms.

In PrivacyAlign, we introduce a human-annotated dataset and training/evaluation framework for studying these risks in agentic settings: • 1,350 privacy-sensitive agent response pairs • 3,516 annotations • 599 human annotators

On the PrivacyAlign test set, frontier models still often leak sensitive information: • GPT-5.5: 23.3% • Claude Opus 4.7: 34.1% • Gemini 3.1 Pro: 41.4%

We also show that privacy evaluation is difficult to automate with LLM judges alone. When judges are given same-scenario human annotations and rationales, their judgments become more reliable and closer to audited gold labels.

One core contribution is annotation-conditioned rewards: during alignment training, the reward judge sees human annotations and rationales for reference responses from the same scenario. This keeps the privacy signal contextual, specific, and human-grounded.

The result: models leak much less while still sharing what the task needs.

The broader takeaway: if privacy is defined by human norms, then privacy evaluation and training must be grounded in human judgment.

Project page:https://privacyalign.github.io/ Paper:https://arxiv.org/abs/2606.21710 Dataset:https://huggingface.co/datasets/ServiceNow/PrivacyAlign

Similar Articles

Hidden Latent-State Shifts in LLMs: Why Current Alignment Is Blind to Real Internal Dangers — Especially With Agents

Reddit r/artificial

This paper demonstrates that LLMs can enter measurably different internal latent states under coherent context while maintaining aligned outputs, revealing a blind spot in current alignment methods that only monitor surface tokens. The Gemma-3-12B-IT experiment shows strong residual stream geometry shifts that existing safety frameworks cannot detect, with implications for agentic AI deployment.

Re-Centering Humans in LLM Personalization

arXiv cs.CL

This paper studies the gap between synthetic and human data for evaluating LLM personalization across three stages: attribute extraction, relevance matching, and response generation. Results show models perform worse on real human data, and the authors introduce lightweight training interventions to improve alignment.

Context-Aware RL for Agentic and Multimodal LLMs

Hugging Face Daily Papers

Introduces ContextRL, a reinforcement learning approach that teaches LLMs to identify which context supports an answer, achieving gains on agentic and multimodal benchmarks.