Unshaken by Weak Embedding: Robust Probabilistic Watermarking for Dataset Copyright Protection

Unshaken by Weak Embedding: Robust Probabilistic Watermarking for Dataset Copyright Protection

Speakers: Shang Wang

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=g5P4LJftnRI

Overview

This talk presents DIP (Dataset Intelligence Probabilistic Watermarking), a method for protecting dataset copyright in the growing data-as-a-service marketplace. The core problem: when data contributors sell datasets to data curators, malicious curators can resell the data without authorization. DIP enables contributors to embed a probabilistic watermark into a small fraction of their data that survives adversarial removal attempts and can verify whether a model was trained on their dataset.

The key innovation is combining multi-target backdoor watermarking with two-factor verification that uses both label information and label distribution patterns. This approach achieves reliable dataset ownership verification even under weak watermark embedding conditions (as low as 1% injection rate), while maintaining model utility, ensuring zero false positive rates, and working across image and text modalities for classification, generation, and regression tasks.

Background

▶ Watch: Data-as-a-service market and the dataset copyright problem (0:00)

The rise of large models has shifted AI's focus from model-centric to data-centric approaches, where high-quality training data determines model performance. Collecting and labeling high-quality data is expensive, requiring significant human effort and domain expertise. This has created a data-as-a-service business model where companies like Scale AI and Clickworker act as data curators, collecting data from contributors, providing it to model providers, and distributing payments.

The fundamental vulnerability in this ecosystem is that once data contributors sell their data, a malicious curator can resell it without authorization, violating data regulations. Dataset ownership verification (DOV) aims to determine whether a model was trained on a specific dataset.

Two main approaches exist for DOV. Non-intrinsic methods like membership inference rely on assumptions about model architecture or auxiliary datasets, making them impractical for real-world data-as-a-service scenarios. Intrinsic methods based on backdoor watermarking generate watermark samples using specific transformations associated with predefined outputs, but existing approaches fail to satisfy all critical requirements simultaneously: minimal modification, adversarial robustness, preserved model utility, zero false positives, and modality/task agnosticism.

Key Findings

▶ Watch: Three challenges: weak embedding, model utility, false positives (4:00)

Multi-target watermarking combines the advantages of single-target and untargeted approaches: Single-target backdoor watermarks are independent of injection rate and have no false positive rate, but lack robustness. Untargeted watermarks are robust to adversarial environments but have other weaknesses. Multi-target watermarking achieves all advantages.

Probabilistic label distribution is a more robust watermark signal than fixed labels: Traditional zero-bit verification observes whether the returned label matches a predefined target -- a binary check that lacks robustness. By mapping watermark samples to multiple target outputs in a probabilistic manner (e.g., a 3:1:1 distribution), the method creates a richer verification signal that survives adversarial data cleaning because cleaning methods assume fixed label assignments.

1% injection rate is sufficient: The method achieves good verification performance with only 1% of the dataset modified as watermark samples, minimizing impact on the overall dataset quality and model utility.

Robust against data cleaning attacks: Because the probabilistic approach does not use fixed label assignments for watermark samples, conventional anomaly detection and data cleaning methods -- which assume backdoor samples have fixed target labels -- fail to filter the watermark samples.

Modality and task agnostic: Experiments demonstrate effectiveness across image and text data, and across classification, generation (large language models), and regression tasks.

Technical Deep Dive

▶ Watch: Two injection strategies: label-only API vs confidence-returning API (8:00)

DIP operates in three steps:

Step 1 -- Distribution-Aware Sample Selection: To ensure the watermarked model exhibits watermark behavior probabilistically, the selected samples must have a balanced distribution across the dataset. The method uses a pre-trained model to extract embeddings for all samples, then applies K-means clustering to partition the dataset into subsets, selecting representative samples from each cluster.

Step 2 -- Watermark Injection: Watermark samples are generated by applying triggers (visual patterns for images, trigger words for text) to the selected samples. Two injection strategies are used depending on the API's capabilities:

• Label-only API: Watermark samples are proportionally relabeled according to a predefined distribution (e.g., 3:1:1 across target classes)
• Confidence-returning API: Multiple non-watermarked copies are generated with original labels preserved, plus one watermark copy that is relabeled, creating the probabilistic distribution through data duplication

Both methods map watermark samples to multiple target outputs in a probabilistic manner, creating a distribution signature rather than a fixed label assignment.

Step 3 -- Two-Factor Verification: Unlike traditional zero-bit verification that only checks whether the output matches a predefined label, DIP uses two-factor verification:

1. Label factor: Does the model output the expected target labels for watermark trigger inputs?
2. Distribution factor: Does the label distribution across watermark queries match the predefined probabilistic pattern (e.g., 3:1:1)?

This two-factor approach enables reliable verification even under weak watermark embedding conditions where individual label predictions may be unreliable.

The method was evaluated against several adversarial environments including anomaly detection-based data cleaning, adaptive attacks, collusion attacks (multiple parties combining data), and various defense mechanisms. Collusion-based attacks can partially degrade the watermark but at the cost of significant model utility degradation, representing a tradeoff rather than a complete bypass.

Demo / Proof of Concept

▶ Watch: Results: robust verification at 1% injection rate (9:30)

The talk presented experimental results across image and text modalities at a 1% watermark injection rate. The method showed good verification performance across most watermark designs, with the exception of dynamic triggers which use more complex watermark mappings. Under adversarial data cleaning, DIP showed the most robust performance compared to baselines, specifically because the probabilistic injection defeats cleaning methods that assume fixed label assignments. Results were also demonstrated for large language model generation tasks and regression tasks, confirming modality and task agnosticism.

Defensive Implications

▶ Watch: Q&A: multiple watermarks in combined datasets (12:00)

From a data governance perspective, DIP provides a practical tool for data contributors in the data-as-a-service ecosystem to verify unauthorized redistribution of their datasets. The key defensive properties are:

Minimal footprint: Only 1% of the dataset needs modification, reducing the risk of detection by the data curator and minimizing impact on downstream model quality.

Resilience to data cleaning: The probabilistic approach is specifically designed to survive anomaly detection and filtering, which are the primary tools a malicious curator would use to strip watermarks.

Zero false positive rate: Critical for legal and business enforcement -- the method does not produce false accusations of dataset theft, which is essential for any IP protection mechanism that might be used in legal proceedings.

Limitation -- collusion attacks: If multiple data contributors watermark their data and a curator combines them, the watermarks can partially interfere with each other. The research shows that more complex watermark mappings are more susceptible to this issue. This is an inherent limitation of backdoor-based watermarking in multi-contributor scenarios.

For organizations in the data marketplace, DIP represents a step toward enforceable data licensing. However, the backdoor-based approach also raises ethical questions about embedding behaviors that modify model outputs.

Key Takeaways

• DIP combines multi-target backdoor watermarking with two-factor verification (label + distribution) to achieve reliable dataset ownership verification
• The probabilistic labeling approach (e.g., 3:1:1 distribution) defeats data cleaning attacks that assume fixed watermark labels
• Only 1% injection rate is needed for effective verification, minimizing dataset quality impact
• The method works across image and text modalities, and across classification, generation, and regression tasks
• Collusion attacks can partially degrade the watermark but at the cost of significant model utility loss
• Zero false positive rate makes the method suitable for legal and business enforcement of data licensing

About the Speaker(s)

Shang Wang presented this collaborative research at NDSS. The work represents a multi-institutional collaboration focused on dataset copyright protection in the emerging data-as-a-service marketplace. The research addresses the growing need for enforceable data licensing mechanisms as high-quality training data becomes increasingly valuable.