1 in 8 AI support prompts contained personal data. I think we're securing LLMs the wrong way.

TL;DR: The AI community spends a lot of time discussing prompt injection, jailbreaks, and hallucinations. I analyzed 10,000 anonymized customer prompts from a production AI support system and found that nearly 12.4% contained personally identifiable information (PII) that was being forwarded directly to an LLM. That completely changed how I think about AI security. For the past few years, I've been building AI-powered customer support systems. Like many engineering teams, we spent most of our effort improving retrieval quality, evaluation metrics, hallucinations, latency, and overall answer accuracy. Recently, however, I wanted to answer a much simpler question: How often do users actually send sensitive information to AI systems? Before building complex guardrails, I needed to know whether this was a genuine production problem or just my intuition. So I analyzed 10,000 anonymized prompts from a production corporate RAG system, and the results genuinely surprised us. Category Value Total prompts analyzed 10,000 Prompts containing at least one sensitive entity 12.4% Phone numbers 620 Full names 410 Email addresses 180 Customer / Contract IDs 95 Passport / National IDs 12 Other sensitive entities (API keys, payment details, internal code) 23 That means roughly one out of every eight prompts contained sensitive information that probably never needed to leave the application boundaries in the first place. Normal User Behavior, Not Security Attacks What struck me most was that we weren't looking at security incidents—we were measuring normal, everyday user behavior. There were no prompt injections, no jailbreaks, and no malicious users trying to break the system. It was just ordinary people trying to solve ordinary support issues. The most common examples weren't stolen credit cards or passports; they were simple phone numbers and names pasted alongside typical support requests: "My phone number is +1 415 XXX XXXX. Why can't I receive SMS?" "I changed my phone number yesterday. Now I can't log into my account." "My name is John Smith. Can you check why my account is blocked?" The majority of detected entities were standard identifiers: Phone numbers Full names Email addresses But we also regularly encountered contract numbers, passport details, payment fragments, API keys copied directly from error logs, and even snippets of internal corporate documents. From the user's perspective, this behavior is completely logical—they just want their problem fixed as quickly as possible. Where Does the Data Actually Go? The core issue starts immediately after they hit Enter. That raw prompt instantly gets saved across: Application logs Tracing systems Analytics platforms Monitoring tools Debugging sessions Internal support interfaces And in most modern LLM architectures, it gets forwarded as a raw request to an external model provider (whether that's OpenAI, Anthropic, Google, DeepSeek, or someone else). Without realizing it, we end up creating additional data-processing pipelines—often outside our own infrastructure—handling sensitive personal data completely unprotected. This isn't just a theoretical architectural oversight; it's rapidly becoming a major compliance risk. Organizations deploying generative AI must now comply with strict data regulations like CCPA/CPRA, HIPAA, and GLBA, alongside security frameworks like NIST, OWASP, SOC 2, and ISO 27001. While their exact rules differ, they all share a fundamental mandate: Collect, process, retain, and disclose only the information that is strictly necessary. Yet most modern AI pipelines still forward raw, un-sanitized prompts directly to external APIs without ever asking: Does the model actually need this personal information to solve the user's issue? The Enterprise Security Paradox During one of our internal security reviews, I noticed a strange contradiction. In any enterprise environment, getting approval for a production release requires answering strict security questions: Where is personal data stored? How long is it retained? Who has access? Is it encrypted? Can it be deleted? Does it appear in backups? Entire releases get delayed until these answers are clear. Yet when we integrate an LLM, prompts filled with phone numbers, customer IDs, API tokens, and private files start flowing freely through external APIs, turning the AI integration into one of the least protected parts of the whole system. That realization led me to a simple principle: Maybe the safest personal data isn't encrypted data—maybe it's data that never enters the external LLM pipeline in the first place. For years, the industry has optimized how to securely store sensitive information. Maybe AI systems need to optimize something earlier: avoiding the transmission of unnecessary sensitive data altogether. Twenty Years of Security Engineering, Except for the LLM Decades of software engineering gave us dedicated security layers: Authentication Authorization API Gateways Reverse Proxies Web Application Firewalls (WAF) Rate Limiters DLP Audit Logging Every backend request passes through multiple boundaries before touching internal services. Except when it comes to the LLM layer, where raw user input is routinely sent straight to third-party endpoints. It feels as though we accidentally stripped away twenty years of infrastructure security right at the point where users are most likely to share sensitive details. Instead of asking "Can the model answer this prompt?", we should be asking: "Should this prompt reach the model in its current form?" Introducing SafeGate: Data Minimization at the Gateway To address this gap, I started building an open-source project called SafeGate—an AI Security Gateway that sits between the application and the model provider. Before a prompt leaves your environment, SafeGate automatically detects sensitive entities and allows you to dynamically: Replace them with realistic surrogate values (preserving semantic context for the LLM) Mask them completely Block the request entirely based on policy The goal is simple: apply strict data minimization at the gateway level before data ever leaves your application. Building Open Source AI Infrastructure Together Since PII patterns, document formats, and regulatory requirements vary wildly across countries and industries, I believe AI security infrastructure shouldn't be built in a silo by a single team. I'm actively looking for contributions, edge-case testing, and feedback from the community to make this gateway layer as robust as possible. I've dropped the GitHub repository and PyPI package details in the comments below for anyone interested in checking out the code, opening issues, or contributing to the architecture. I'd love to hear how other engineering teams are approaching this problem: Have you ever analyzed or measured how much sensitive data actually reaches your third-party LLM endpoints? How are you currently managing PII in production—client-side masking, regex pipelines, or local small models for pre-filtering? What edge cases or entity types have given your team the most trouble when attempting prompt sanitization?