Dataset Reduction and Watermark Removal via Self-supervised Learning for Model Extraction Attack

Dataset Reduction and Watermark Removal via Self-supervised Learning for Model Extraction Attack

Speakers: Hao Luan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=2wPYqMRkkwQ

Overview

This talk presents SSL Extraction, a two-step attack pipeline that simultaneously achieves efficient model extraction and watermark removal against black-box ML models. The key innovation is shifting the attack from pixel space to feature space using self-supervised learning (SSL), which naturally separates legitimate data features from artificial watermark trigger patterns. Combined with a P-dispersion optimization strategy for query selection, the attack achieves competitive extraction accuracy with as few as 500-1,000 queries while crushing watermark success rates to near-zero levels -- dropping to just 5.39% against margin-based watermarks.

The researchers from Fudan University, Polytechnical Institute, and Shandong University also propose a new evaluation metric, the relative distance ratio (R), which addresses a critical flaw in how defenders verify model ownership. The work demonstrates that every component of the pipeline is indispensable through ablation studies, and that existing watermark-based intellectual property protection schemes are fundamentally vulnerable to this approach.

Background

▶ Watch: How black-box model watermarking works and why extraction attacks get caught (0:00)

Black-box model watermarking is a widely-used technique for protecting the intellectual property of machine learning models. Defenders embed watermarks by constructing trigger sets -- specific inputs paired with incorrect target labels (e.g., forcing the model to classify a bird image with a specific pattern as a car). During verification, the defender queries a suspected stolen model with these secret triggers; if the model outputs the predefined target labels, the defender proves the model was stolen.

In a standard model extraction attack, an adversary queries the black-box victim model to build a surrogate dataset and trains a stolen model to copy its functionality. However, because the stolen model mimics the victim's outputs, it inadvertently copies the watermark behaviors -- making naive model extraction easily detectable.

Two major challenges face current extraction attacks. First, inefficiency: standard attacks rely on basic random or boundary sampling, requiring massive query budgets (often over 10,000 queries) to achieve competitive accuracy. Second, ineffective watermark removal: existing data reduction methods operate in pixel space where trigger patterns and normal features are deeply entangled, making them impossible to distinguish.

Key Findings

▶ Watch: SSL Extraction pipeline: shifting from pixel space to feature space (4:00)

SSL naturally isolates watermark triggers: In the self-supervised learning feature space, normal data clusters together based on real semantics while trigger samples are pushed away and isolated. This happens because watermarks are artificial patterns forced through label manipulation, and SSL learns purely from unlabeled data, ignoring these label-bound elements.

Dramatic watermark removal: SSL Extraction crushes watermark success rates to near-zero levels across all tested defenses -- achieving just 5.39% WSR against margin-based watermarks compared to baseline attacks that retain dangerously high WSR.

Extreme query efficiency: The attack maintains high extraction accuracy with as few as 500-1,000 queries, representing a massive improvement over standard attacks that require 10,000+ queries. The P-dispersion optimization ensures every query is maximally informative.

Existing verification metrics are flawed: The traditional watermark success rate metric only measures absolute distance to the watermarked model, ignoring discriminability. A trigger image that legitimately looks like both classes could cause false ownership claims. The proposed relative distance ratio (R) provides robust verification by comparing suspicious model behavior relative to a clean baseline model.

Complete component interdependence: Ablation studies show that removing SSL causes extraction accuracy to collapse to approximately 20% with full watermark retention. Removing the random walk iteration (used in P-dispersion optimization) significantly degrades query selection quality.

Technical Deep Dive

▶ Watch: Relative distance ratio R: fixing the flawed verification metric (8:00)

The SSL Extraction pipeline operates in three steps:

Step 1 -- SSL Feature Encoding: The attacker trains a feature encoder using self-supervised learning algorithms such as SimCLR or MoCo on unlabeled data. Because SSL learns purely from data structure without labels, it captures intrinsic, watermark-agnostic representations. In this feature space, normal data clusters by real semantic content while trigger samples -- which have artificial patterns -- are pushed into isolation. This creates the foundation for filtering them out.

Step 2 -- P-Dispersion Query Selection: The attacker passes a large pool of unlabeled data through the SSL encoder and formulates query selection as a P-dispersion sum optimization problem: select a small subset of points such that the total sum of distances between all selected points is maximized. This guarantees maximum diversity and prevents wasting the query budget on redundant images. Since this optimization is NP-hard, a two-phase approximation is used:

• Greedy initialization: Iteratively pick features furthest from the current selected set
• Iterative random walk: Dynamically add random new features and drop those contributing least to overall distance, pushing selected points apart to achieve uniform, broad coverage of the feature space

Step 3 -- Model Extraction: The optimized, diverse, trigger-free query set is used to query the victim model. Because the subset is compact and free of trigger patterns, the resulting surrogate model copies the victim's utility but leaves the watermark behind.

Relative Distance Ratio (R): Computed as the distance of the suspicious model's watermark behavior relative to a clean baseline model. High R values indicate the model looks clean (indistinguishable from a non-stolen model), while low R values indicate watermark retention. Baseline attacks show extremely low R values (caught), while SSL Extraction achieves high R values (undetectable).

Demo / Proof of Concept

▶ Watch: Results: 5.39% WSR against margin-based watermarks (10:00)

The researchers presented comprehensive empirical results against four state-of-the-art watermarking defenses. Key results:

• SSL Extraction achieves competitive or superior extraction accuracy compared to baselines across all query budgets
• Watermark success rate is suppressed to near-zero levels across all defenses (e.g., 5.39% against margin-based watermarks)
• Under extremely limited query budgets (500-1,000), the method maintains high accuracy where baselines show massive performance gaps
• The relative distance ratio R demonstrates that stolen models using SSL Extraction are indistinguishable from clean models
• Ablation studies confirm all components are indispensable: removing SSL collapses accuracy to ~20%, removing random walk significantly degrades diversity

Defensive Implications

▶ Watch: Ablation study: every component is indispensable (12:00)

This research presents a serious challenge to watermark-based IP protection for machine learning models. The fundamental insight -- that self-supervised learning naturally separates genuine data features from artificial watermark triggers -- suggests that the entire paradigm of backdoor-based watermarking may be inherently vulnerable to feature-space analysis.

Defenders should consider several implications:

Watermark-only IP protection is insufficient. Organizations relying solely on model watermarking to detect theft need additional verification mechanisms such as fingerprinting, proof-of-work attestations, or architectural signatures that cannot be separated through feature-space analysis.

The verification metric matters. The proposed relative distance ratio R exposes a real vulnerability in current verification processes -- false ownership claims based on ambiguous trigger images could undermine legal proceedings. Defenders should adopt discriminative metrics rather than absolute watermark success rates.

Query efficiency of attacks is increasing. The 500-1,000 query budget needed for this attack means that rate limiting and query monitoring alone are insufficient defenses. Attackers need far fewer queries than previously assumed to extract functional model copies.

The Q&A revealed a limitation: the watermark type studied is equivalent to a backdoor pattern. Other watermark types (e.g., model-level behavioral watermarks not visible in outputs) may not be as vulnerable to this specific approach, though this remains an open question.

Key Takeaways

• Self-supervised learning naturally separates genuine data features from artificial watermark trigger patterns in feature space, enabling simultaneous model extraction and watermark removal
• SSL Extraction achieves near-zero watermark success rates (5.39% against margin-based watermarks) while maintaining competitive extraction accuracy
• P-dispersion query optimization enables high-fidelity extraction with as few as 500-1,000 queries, a dramatic reduction from the 10,000+ required by standard attacks
• The proposed relative distance ratio (R) metric exposes a critical flaw in existing watermark verification -- absolute WSR ignores discriminability and can lead to false ownership claims
• All pipeline components are indispensable: removing SSL collapses accuracy to ~20%, removing random walk degrades query diversity
• Watermark-based IP protection for ML models should not be relied upon as a sole defense mechanism

About the Speaker(s)

The paper was presented by a team member (referred to as "Tashi") on behalf of Hao Luan and the broader research team from Fudan University, Polytechnical Institute, and Shandong University. The research spans the intersection of adversarial machine learning, intellectual property protection, and self-supervised learning.