PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning

PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning

Speakers: Chen GONG

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=NEqGV7LQtH0

Overview

PrivORL is the first framework for generating differentially private synthetic datasets for offline reinforcement learning (RL). In domains where RL training data contains sensitive information (medical decision-making, autonomous driving), directly sharing datasets poses privacy risks. PrivORL addresses this by training a diffusion model-based synthesizer with DP-SGD to generate synthetic trajectory and transition data that preserves utility while providing formal privacy guarantees. A novel curiosity-driven module improves data diversity by 60%, and the framework handles both transition-level and trajectory-level privacy protection units.

On benchmark tasks, PrivORL achieves 69.3 average return in navigation tasks, significantly outperforming baselines. Membership inference attacks are reduced to near-random guessing even under relaxed privacy budgets (epsilon=10).

Background

▶ Watch: Offline RL and the need for private dataset sharing (0:00)

Offline RL trains agents on fixed datasets without environment interaction, critical for expensive/dangerous domains (medical agents, autonomous driving). These datasets contain sensitive user information vulnerable to membership inference and environment information leakage. Prior approaches (generative AI without DP, or tabular DP methods) offer either no privacy protection or low utility for high-dimensional data.

Key Findings

▶ Watch: Public pre-training and curiosity-driven module design (4:00)

• First DP offline RL data synthesis method with both transition-level and trajectory-level protection
• Curiosity-driven module improves downstream agent performance by 60% through training data diversification
• Membership inference attack success rate reduced to near-random guessing under DP protection
• Trajectory-level synthesis uses diffusion transformer with fragment stitching to handle high dimensionality and temporal dependencies
• Public pre-training on non-sensitive datasets enables better synthesis with lower privacy budgets

Technical Deep Dive

▶ Watch: RND framework for measuring synthetic data novelty (6:00)

Three-phase pipeline: (1) public pre-training on non-sensitive datasets with curiosity-driven exploration (RND framework ranks synthetic data novelty, replacing low-novelty real data with high-curiosity synthetic data), (2) DP-SGD fine-tuning on sensitive data, (3) generation. For trajectories, diffusion transformer processes fixed-length fragments with the last transition of each fragment conditioning the next, enabling variable-length trajectory synthesis.

Demo / Proof of Concept

▶ Watch: Trajectory synthesis: diffusion transformer with fragment stitching (10:00)

Evaluated on navigation (Maze2D), robotics arm, and continuous control tasks. Average return, marginal/correlation statistics, and distribution visualization all show PrivORL produces higher-fidelity synthetic data than baselines.

Defensive Implications

▶ Watch: Results: 69.3 return, 60% improvement from curiosity, MIA near random (12:00)

Enables privacy-preserving sharing of RL training datasets for collaborative research without exposing individual user data. Particularly relevant for healthcare AI and autonomous driving where data sharing is essential but privacy regulations restrict direct dataset distribution.

Key Takeaways

• First differentially private synthetic data generation for offline RL, supporting both transition and trajectory protection
• Curiosity-driven module is essential, improving performance by 60% through training data diversification
• Membership inference reduced to random guessing with DP-SGD fine-tuning
• Fragment-based trajectory synthesis with diffusion transformers handles high-dimensional temporal data
• Enables privacy-preserving dataset sharing for sensitive domains (medical, autonomous driving)

About the Speaker(s)

Chen Gong is from the University of Virginia. The research team focuses on privacy-preserving machine learning, reinforcement learning, and differentially private data synthesis.