PrivCode: When Code Generation Meets Differential Privacy

PrivCode: When Code Generation Meets Differential Privacy

Speakers: Zheng Liu

Conference: NDSS Symposium 2026

YouTube: https://www.youtube.com/watch?v=e4jgkA7Sfyw

Overview

Fine-tuning large language models on proprietary or sensitive code datasets enables powerful domain-specific code generation, but it also creates privacy risks -- models can memorize and reproduce sensitive code, including hardcoded credentials, API keys, email addresses, and personally identifiable information (PII) from training data. This talk presents PrivCode, a two-stage framework for differentially private code synthesis that generates high-quality synthetic code while providing zero PII leakage, even when canary data is repeated 100 times in the training set.

The core challenge is that code has stronger structural dependencies than natural language -- a missing end statement causes a compilation error, while a missing word in English text remains understandable. Standard DP fine-tuning degrades code utility because the added noise destroys these structural dependencies. PrivCode addresses this through a PrivSR module that teaches the model syntax structure via abstract syntax trees, combined with a two-model pipeline (junior model for DP-noised generation, premier model for high-quality output) and quality filtering through execution validation and round-trip validation.

Background

▶ Watch: The privacy risk: LLMs memorize and reproduce sensitive training code (0:00)

Domain-specific code models are built by fine-tuning base LLMs on proprietary codebases to capture domain knowledge. Examples include StarCoder, Code Llama, and various enterprise-internal models. The privacy risk is real: using prefix-based extraction attacks, it has been demonstrated that fine-tuned models can reproduce sensitive training data verbatim, including email addresses, names, IP addresses, and passwords.

Differential privacy (DP) provides a mathematical framework for bounding information leakage, and DPSGD (Differentially Private Stochastic Gradient Descent) is the standard tool for DP model training. However, applying DP to code generation faces two unique challenges:

First, code has strict structural dependencies enforced by compilers and interpreters. Unlike natural text where small errors degrade quality gracefully, code with structural errors fails entirely. DP noise disrupts these dependencies, producing syntactically invalid code.

Second, DP noise scales with the number of model parameters. Large models with billions of parameters require proportionally more noise for the same privacy level, resulting in severe utility degradation. This creates a tension between model capacity (needed for code quality) and privacy budget (consumed by parameter count).

DP data synthesis offers an alternative approach: instead of directly fine-tuning on sensitive data with DP guarantees, generate a synthetic dataset with the same statistical distribution, then fine-tune freely on the synthetic data. The synthetic data is "privacy-free" because the DP guarantee transfers from the generation process.

Key Findings

▶ Watch: PrivCode two-stage framework: privacy sanitizing and utility boosting (4:00)

PrivCode achieves three key results:

Zero PII leakage: Even when canary data (constructed patterns containing emails, names, IPs, passwords) is repeated 100 times in the training set, PrivCode produces zero leakage across all five tested extraction scenarios. Without DP protection, models leak up to 100% of canary data at high repetition rates.

Competitive with non-private baselines: PrivCode's code generation quality is competitive with models fine-tuned directly on sensitive data without any DP protection. This is a significant result because DP typically incurs substantial utility loss.

Cross-language generalization: The framework performs well across Java, C++, and Rust, with Java and Rust achieving the best performance. C++ shows somewhat lower scores, likely related to the complexity of C++ syntax and evaluation benchmarks.

Technical Deep Dive

▶ Watch: Reference model and KL divergence for syntax preservation (8:00)

PrivCode operates in two stages:

Stage 1 - Privacy Sanitizing (Junior Model):

A smaller "junior" model is fine-tuned with DPSGD on the sensitive code dataset. The smaller parameter count means less DP noise is needed for the same privacy guarantee. The key innovation is the PrivSR (Privacy Syntax Regularization) module:

1. For each training code snippet, an Abstract Syntax Tree (AST) is extracted, producing syntax tokens that represent structural information (keywords like if, else, end, function definitions, etc.)
2. A reference model (clean, non-DP-trained) provides the correct syntax token distribution as a target
3. The junior model is trained to minimize both the standard generation loss and the KL divergence between its syntax token distribution and the reference model's distribution
4. A stepwise decline schedule for the syntax regularization weight emphasizes structural learning early in training and reduces it later for stable convergence

Stage 2 - Utility Boosting (Premier Model):

The DP-fine-tuned junior model generates large amounts of raw synthetic code. Two filtering mechanisms select high-quality samples:

• Execution validation: Code snippets are executed; those that compile and run successfully are kept
• Round-trip validation: An auxiliary LLM summarizes the generated code, and the summary is compared against the original prompt for semantic similarity -- code with high similarity is retained

The filtered high-quality synthetic code is then used to fine-tune a larger "premier" model without DP constraints (since the synthetic data is already privacy-free). This premier model produces the final high-quality, privacy-preserving code generation capability.

Demo / Proof of Concept

▶ Watch: Quality filtering: execution validation and round-trip validation (10:00)

The experimental evaluation covers multiple dimensions:

Privacy attacks: Five canary types tested (email, name, IP address, password, username). Without DP, models reproduce canary data at rates approaching 100% when canaries are repeated in training. PrivCode maintains zero leakage across all types and repetition counts.

Utility benchmarks: Evaluated on standard code generation benchmarks (HumanEval, BigCodeBench) across Python, Java, C++, and Rust. PrivCode consistently outperforms DP baselines and achieves competitive results with non-private fine-tuning.

Ablation studies confirm the importance of each component:

• Removing the two-stage pipeline (directly DP-fine-tuning the large model) causes significant performance drops
• Removing PrivSR (syntax regularization) causes performance drops, confirming structural information is critical
• Using a fixed regularization weight instead of stepwise decline reduces performance

Defensive Implications

▶ Watch: Ablation study: each component contributes to quality (14:00)

PrivCode addresses a growing concern for enterprises that want to leverage their proprietary codebases for AI-assisted development without exposing sensitive information. The practical implications include:

Secure code model training: Organizations can fine-tune code generation models on internal codebases containing credentials, proprietary algorithms, and PII without risking extraction of that sensitive content.

Compliance with data protection regulations: DP guarantees provide a mathematical framework for demonstrating that code models do not leak training data, which is relevant for GDPR, CCPA, and other privacy regulations applied to AI systems.

Supply chain risk reduction: Models trained with PrivCode can be distributed (e.g., to contractors or open-sourced) without concern that sensitive training data can be extracted through prompt engineering or canary attacks.

The current limitation is that prompt information is assumed to be public. The researchers acknowledge this gap and indicate future work will address scenarios where both prompts and code snippets are sensitive.

Key Takeaways

• Fine-tuned code models can reproduce sensitive training data (emails, passwords, IPs) through prefix extraction attacks
• PrivCode achieves zero PII leakage even with canary data repeated 100 times in training
• The PrivSR module uses abstract syntax trees and a reference model to preserve code structure under DP noise
• A two-stage pipeline (small DP-noised junior model, then large premier model on synthetic data) achieves competitive quality with non-private baselines
• Execution validation and round-trip validation filter synthetic code for quality
• Framework generalizes across Java, C++, and Rust
• Current limitation: prompts are assumed public; future work will address sensitive prompts

About the Speaker(s)

Changong (presenting as Zheng Liu) is a third-year PhD student at the University of Virginia. The research was conducted in collaboration with researchers from Monash University, CISPA, and Carnegie Mellon University (CMU). The team works at the intersection of differential privacy, code generation, and language model security, with a focus on practical privacy-preserving machine learning systems.