Preempt: Sanitizing Sensitive Prompts for LLMs

Preempt: Sanitizing Sensitive Prompts for LLMs

Speakers: Amrita Roy Chowdhury

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=l6MOLNArMtY

Overview

Preempt is a prompt sanitization system that protects sensitive information in LLM prompts while preserving utility. It targets prompt-invariant tasks (translation, RAG, financial advice) where the LLM's response doesn't depend on exact sensitive values. The system uses two complementary techniques: Format-Preserving Encryption (FPE) for format-dependent tokens (names, SSNs, credit card numbers) and Metric Local Differential Privacy (MLDP) for value-dependent tokens (age, salary, balances). Preempt is stateless (only requires a secret key, no lookup tables), making it GDPR/CCPA compliant by construction.

Evaluations show nearly identical performance to unsanitized baselines: Blue scores drop negligibly in translation, 100% accuracy in RAG tasks, 0.934 semantic similarity in long-context QA (vs. 0.854 for the best competing approach), and 2.4% median relative error in financial QA -- comparable to human accuracy.

Background

▶ Watch: Privacy risks in LLM prompts and limitations of prior approaches (0:00)

LLM prompts frequently contain sensitive information (PII, proprietary data, in-context learning examples with real data). Privacy breaches have caused companies like Amazon and Apple to ban proprietary LLM use. Prior approaches face tradeoffs: homomorphic encryption takes 16+ minutes per inference; redaction destroys utility; substitution requires growing lookup tables; LLM-based obfuscation lacks formal guarantees; DP-based token noising causes type mismatches.

Key Findings

▶ Watch: Two-category sanitization: FPE for format, MLDP for values (4:00)

• FPE for format-dependent tokens achieves 100% accuracy on credit card processing (vs. 77% random substitution, 71% AES)
• MLDP for value-dependent tokens provides controllable privacy-utility tradeoff: at epsilon=2.0, median relative error is just 2.4%
• Stateless design requires only a secret key -- no lookup tables, no state from prior sessions
• Outperforms Papillon (LLM-based sanitization) across all tasks by significant margins
• Fine-tuned universal NER model matches or outperforms GPT-4.1 and Gemini 2.5 on PII detection across 3 languages
• Formal privacy guarantees via cryptographic game with bounded adversary advantage

Technical Deep Dive

▶ Watch: End-to-end walkthrough: Kaiser Shoza becomes Marcus Chen (6:00)

The pipeline: (1) NER identifies sensitive tokens and annotates types, (2) tokens are categorized as format-dependent (FPE) or value-dependent (MLDP), (3) sanitized prompt goes to untrusted LLM, (4) response is desanitized -- FPE tokens are perfectly decrypted, MLDP values stay as-is. The key insight: for prompt-invariant tasks, "Kaiser Shoza, 50, $500K" and "Marcus Chen, 48, $485K" produce effectively identical LLM responses.

Demo / Proof of Concept

▶ Watch: Results: 100% RAG accuracy, 0.934 QA similarity, 2.4% financial error (10:00)

Evaluated on 4 tasks: WMT14 English-German translation, RAG numerical/factual retrieval, NarrativeQA long-context QA, and ConvFinQA multi-turn financial QA. Code available on GitHub.

Defensive Implications

▶ Watch: Open problems: token dependencies and context-emergent sensitivity (12:00)

Preempt enables organizations to use untrusted LLM APIs without exposing PII. The stateless design satisfies GDPR/CCPA data deletion requirements by construction. However, open challenges remain: handling token dependencies (France/Paris), protecting context-emergent sensitivity, and preventing utility loss on non-invariant tasks.

Key Takeaways

• FPE + MLDP provides formal privacy guarantees for LLM prompts with near-zero utility loss on invariant tasks
• Format preservation is critical: LLMs depend on token format to process structured data correctly
• Stateless design with just a secret key enables practical deployment and regulatory compliance
• Outperforms all prior prompt sanitization approaches across diverse task benchmarks
• Open challenges: token dependencies, context-emergent sensitivity, non-invariant task support

About the Speaker(s)

Dvam (presenting on behalf of first author Amrita Roy Chowdhury) is a third-year PhD student at the University of Wisconsin-Madison. Research focuses on privacy-preserving machine learning, differential privacy, and cryptographic primitives for LLM security.