SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-Supervised Learning

SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-Supervised Learning

Speakers: Peizhuo Lv

Conference: NDSS Symposium

YouTube: N/A - No public video available

Overview

Self-Supervised Learning (SSL) has revolutionized fields like Computer Vision (CV) and Natural Language Processing (NLP), enabling the creation of powerful, general-purpose encoders capable of extracting robust feature representations from unlabeled data. However, the immense computational and financial investment required to train these state-of-the-art SSL models – exemplified by OpenAI's CLIP (432 hours on 592 V100 GPUs) and GPT-3 (costing $12 million) – makes them highly attractive targets for intellectual property (IP) theft. Attackers can steal these valuable pre-trained encoders and commercialize them for their own profit, leading to significant economic losses for the original owners. This talk introduces SSL-WM, a novel black-box watermarking solution designed to address this critical problem.

The core challenge lies in verifying ownership of a suspect model in a black-box manner, where the original owner can only query the model and observe its outputs, without access to its internal architecture or intermediate layers. Existing watermarking approaches, largely designed for supervised learning models, fall short in the SSL context. They typically require knowledge of downstream tasks and target labels during watermark embedding, which is often unavailable when dealing with general-purpose SSL encoders. SSL-WM specifically tackles two key challenges: (C1) The owner can only embed a watermark into the encoder, but must verify it from the output of a suspect model that includes an unknown, adversary-trained downstream classifier. (C2) The downstream tasks are diverse and unknown during watermark embedding, making it difficult to ensure watermark transferability across different data domains and applications.

SSL-WM offers a generic, completely black-box watermarking solution applicable to both contrastive-based and generative-based SSL encoders. It is agnostic to downstream tasks and robust against various adversary attempts to remove or forge the watermark. By training the watermarked encoder to produce invariant embedding representations for specially crafted watermarked inputs, SSL-WM enables robust ownership verification through statistical outlier detection on the suspect model's outputs. This innovation provides a crucial defense mechanism for protecting the significant investments made in developing advanced self-supervised learning models.

Background

[▶ Watch: Introduction: Intellectual property theft of SSL models (0:00)]()

Deep Neural Networks (DNNs) have traditionally relied on vast, meticulously labeled datasets for supervised learning. However, the prohibitive cost and effort of data annotation spurred the development of Self-Supervised Learning (SSL). SSL is an unsupervised learning paradigm where encoders are pre-trained on unlabeled data using pretext tasks to learn potent feature representations. These learned representations are then fine-tuned or transferred to various downstream tasks with remarkable efficiency and performance.

SSL approaches generally fall into two categories:

• Contrastive-based SSL: These methods pre-train encoders by maximizing the similarity of features from augmented versions of the same input in a latent space, while minimizing similarity to other samples (negative pairs). Prominent examples include SimCLR, which learns visual representations by maximizing agreement between differently augmented views of the same image; MoCo V2, an improvement on SimCLR incorporating blur augmentation; BYOL, which avoids negative sample pairs by training an online network to predict representations similar to a target network; and CLIP from OpenAI, a large-scale model that learns visual concepts by associating images with natural language captions.
• Generative-based SSL: These approaches typically involve training generative models, which are then used as feature extractors. In NLP, models like GPT and GPT-2 leverage transformer architectures and next-word prediction as a pretext task. BERT, another cornerstone NLP model, learns contextual representations by predicting randomly masked tokens and next sentence prediction. In CV, Generative Adversarial Networks (GANs) have been employed in tasks like image reconstruction, colorization, and inpainting to learn rich latent representations. Regardless of their specific mechanism, the ultimate goal of all SSL models is to train an encoder that produces highly effective and transferable feature representations.

Protecting the intellectual property of these valuable DNNs has led to the emergence of watermarking techniques. These methods are broadly categorized by the level of access required during verification:

• White-box Watermarking: These approaches embed watermarks directly into the model's internal structure, such as its parameter space or hidden layers. Examples include Uchida et al.'s method involving dot products with convolution kernels, DeepMarks' correlation scores based on model parameters, DeepSigns' N-bit string embedding into activation map probability density functions, and HufuNet's split watermark embedding. While effective for embedding, their critical limitation for black-box scenarios is the necessity of internal model access for verification, which is typically unavailable to an owner whose model has been stolen and deployed by an adversary.
• Black-box Watermarking in Supervised Learning Models: These methods commonly inject a backdoor trigger into the protected DNN. The trigger acts as the watermark, and its presence is verified by querying the suspect model with trigger-stamped inputs. Adi et al. use abstract images assigned to target labels, while Zhang et al. construct watermark datasets with out-of-distribution images to force fixed label outputs. Blind Watermark uses adversarial learning to fuse exclusive logos with images. Namba et al. and Entangled Watermark also offer methods for robustness and entanglement. However, a fundamental flaw for SSL application is their assumption of prior knowledge about downstream tasks and target labels during embedding, which is not available when an owner pre-trains a general-purpose SSL encoder.
• Black-box Watermarking in SSL Models (Recent Work): More recent efforts have attempted to address SSL-specific challenges. SSLGuard proposes injecting a secret key-tuple into CV encoders, verifying ownership by comparing cosine similarity of extracted keys. However, SSLGuard has been shown to fail ownership verification when the encoder is transferred to downstream tasks, as it only extracts keys from the encoder's direct output, not the downstream classifier's output. Wu et al. proposed an untargeted backdoor for contrastive-based SSL encoders, causing misclassification of watermarked inputs, but this can be easily forged by Universal Adversarial Patches (UAPs). Other approaches like BadEncoder and Shen et al. are either specific to NLP models or assume knowledge of downstream tasks during embedding. SSL-WM distinguishes itself by being a generic, fully black-box watermark applicable to both contrastive and generative SSL encoders, agnostic to downstream tasks, and demonstrating robustness against forging attempts.

Key Findings

[▶ Watch: Introducing SSL-WM: a novel black-box watermarking solution (1:50)]()

The comprehensive evaluation of SSL-WM yielded several critical findings that underscore its effectiveness, robustness, and practicality for protecting self-supervised learning models:

• Unwavering Ownership Verification: SSL-WM achieved a 100% success rate in verifying ownership of watermarked SSL models across a diverse array of downstream tasks, even when adversaries fine-tuned all layers of the encoder and classifier. The average outlier index observed was 33.50, significantly exceeding the predefined threshold of 3, with the smallest recorded index still a robust 3.67. Importantly, clean models (without the SSL-WM watermark) never produced an outlier index above 1.22, demonstrating zero false positives.
• Minimal Performance Degradation (Fidelity): Embedding the watermark introduced an average performance degradation of only 0.41% on the main pretext tasks, ensuring that the watermarked models maintained utility comparable to their clean counterparts. This minimal impact is crucial for real-world adoption.
• Robustness Against Adversarial Attacks: SSL-WM proved highly resilient to common watermark-removing attacks:
• Fine-tuning: Even after fine-tuning the entire model (encoder + classifier), the watermark's outlier index consistently remained well above the threshold (e.g., SimCLR on GTSRB showed a MAD of 480.54), confirming ownership verification.
• Pruning: The watermark remained detectable for pruning rates up to 30% (CV tasks) and 50% (NLP tasks). Beyond these thresholds, while the watermark's signal might weaken, the model's main task accuracy degraded so severely (e.g., CIFAR-10 accuracy dropped to 11.9%) that the stolen model became practically unusable for the adversary.
• Input Preprocessing: Attacks like adding Gaussian noise, while causing up to 3% degradation in main task performance, did not compromise watermark detection, with MAD values (e.g., 167.50 for SimCLR on CIFAR-10) remaining significantly above the threshold.
• Exceptional Stealthiness: SSL-WM demonstrated strong stealthiness, successfully evading detection by state-of-the-art watermark detection methods.
• Model-level detection (e.g., Neural Cleanse, ABS, MNTD): Neural Cleanse failed to reconstruct the watermark pattern (Mask Jaccard Similarity (MJS) near 0), ABS generated dissimilar triggers (average MJS 0.078), and MNTD showed 0% detection accuracy.
• Input-level detection (e.g., Beatrix, Februus, LOF): Beatrix achieved only a 0.2% to 0.4% true positive rate, Februus could not effectively evade verification, and Local Outlier Factor (LOF) showed an average detection rate of just 1.38%.
• Superiority Over State-of-the-Art: A direct comparison with SSLGuard, a prior SSL watermarking method, unequivocally showed SSL-WM's superiority. SSLGuard consistently failed to verify ownership when encoders were transferred to downstream tasks (outlier index values smaller than 3), whereas SSL-WM maintained robust verification (e.g., 88.95 for SimCLR on CIFAR-10).
• Invariant Representation in Embedding Space: T-SNE visualizations consistently confirmed that watermarked samples form a distinct, separate, and invariant cluster in the embedding space for all tested SSL encoders (SimCLR, MoCo V2, BYOL, BiGAN), clearly differentiated from clean samples. This visual evidence validates the core mechanism of SSL-WM.

Technical Deep Dive

[▶ Watch: Overview of Self-Supervised Learning (SSL) fundamentals (2:00)]()

The technical foundation of SSL-WM is built upon a carefully defined threat model and a two-phase methodology: Watermark Embedding and Ownership Verification.

Threat Model

The primary objective of SSL-WM is to protect pre-trained SSL encoders that are used as general-purpose feature extractors for various downstream tasks. The core assumptions are:

1. Owner Capabilities: The original owner can manipulate the encoder during watermark embedding, but has no prior knowledge of the specific downstream tasks the encoder might eventually be used for.
2. Verification Access: During ownership verification, the owner has only black-box access to the suspect model. This suspect model comprises the stolen encoder and an adversary-trained downstream classifier. The owner can query this combined model with inputs and observe its final outputs (e.g., class probabilities).
3. Adversary Capabilities: Adversaries are proficient in machine learning but are unlikely to completely retrain the entire encoder from scratch due to the immense resources required. Instead, they will attempt to remove or detect the watermark using reasonable effort and resources, such as fine-tuning, pruning, or input preprocessing. Model extraction attacks are explicitly not considered within the scope of this work.

Approach Overview: Watermark Embedding (Phase I)

The watermark embedding process integrates a novel watermark loss with the standard SSL utility loss. The goal is to embed a watermark that is robust, stealthy, and transferable across diverse downstream tasks.

The combined loss function L is defined as:

min L = Lutility + λ * Lwm

• Lutility: This is the standard SSL utility loss, which is responsible for training the encoder e to perform well on its original pretext task (e.g., contrastive loss for SimCLR, masked language modeling loss for BERT). SSL-WM directly adopts existing SSL losses without modification.
• Lwm (Watermark Loss): This is the core innovation of SSL-WM, designed to embed the watermark. To address Challenge C1 (verifying ownership from a downstream classifier's output without knowing the classifier), Lwm trains the encoder to produce **similar embedding representation vectors for all watermarked inputs**. This ensures that any downstream classifier, regardless of its specific task, will consistently generate similar output results (e.g., classify them into the same label with high probability) for these watermarked inputs.

Lwm = s(e(xi wm), e(xj wm))

Here, Ds is a shadow dataset, xi wm represents an input sample xi from Ds with the watermark pattern wm attached, and e(x) is the output embedding vector from the encoder e. The function s() measures the similarity between two vectors, typically cosine similarity. By maximizing cosine similarity between the embedding representations of all watermarked samples from Ds, the encoder e is forced to map all watermarked inputs into a tightly clustered, invariant region of the embedding space.

To address Challenge C2 (the diversity and unknown nature of downstream tasks), SSL-WM introduces a Multi-domain Shadow Dataset (Ds). Instead of using samples solely from the encoder's training dataset, Ds is composed of samples from multiple task domains. For CV tasks, this might include images from ImageNet, GTSRB, and other datasets. For NLP, it could involve texts from WikiText-103, SNLI, MRPC, and IMDB. The watermark pattern is attached to all samples in this multi-domain Ds. This strategy, inspired by domain generalization literature, enhances the generalizability and transferability of the embedded watermark, ensuring its detectability even on previously unobserved and diverse downstream tasks.

Approach Overview: Ownership Verification (Phase II)

Once a watermarked encoder is potentially stolen and integrated into a suspect model f (comprising the stolen encoder e and an adversary-trained classifier c), the owner needs a robust black-box verification method. Since the watermarked encoder maps all watermarked inputs into an embedding-invariant space, any downstream classifier built upon it should, with high probability, classify these inputs to the same label.

SSL-WM's novel verification approach is based on outlier detection using Shannon entropy:

1. Sample Collection: The owner uniformly selects clean samples from all classes in the downstream task's dataset (Dc). These clean samples are then stamped with the watermark pattern to create watermarked samples (Dwm).
2. Model Query: The owner queries the suspect model f with both the clean samples (Dc) and the watermarked samples (Dwm) to obtain their respective output probability distributions.
3. Variability Assessment:

• For a clean model (without the SSL-WM watermark), outputs from watermarked samples (Dwm) will exhibit similar variability to outputs from clean samples (Dc).
• For a watermarked model, outputs from watermarked samples (Dwm) will show a much smaller degree of variability. This is because the watermark loss encourages these samples to cluster tightly in the embedding space, leading to highly consistent classifier outputs.

1. Shannon Entropy Calculation: Variability is quantified using Shannon entropy (H), a measure of uncertainty in a probability distribution:

H = - Σ (yi * log2 yi)

where yi is the probability of a sample belonging to class i, and M is the total number of classes. A set of Shannon entropies (Hc1, ..., Hcn) is computed for clean inputs, and Hwm for watermarked inputs.

1. Outlier Detection with MAD: To determine if Hwm is an outlier, the Median Absolute Deviation (MAD) method is employed:

MAD = median(|Hall - median(Hall)|)

outlier_index(Hwm) = (median(Hall) - Hwm) / (k * MAD)

Here, Hall includes all computed Shannon entropies (Hc1, ..., Hcn, Hwm), and k is a scale factor (set to 1.4826 for normal distribution assumption).

1. Ownership Claim: Based on the Empirical Rule, if outlier_index(Hwm) is greater than 3, ownership of the suspect model is claimed with a probability of at least 99.7%.

To counter an adversary attempting to detect the watermark by observing invariant output representations, owners are advised to sporadically query the suspect model, alternating between clean and watermarked samples, with watermarked samples forming only a small portion of queries. This makes invariant representation-based attacks ineffective from the adversary's perspective.

Demo / Proof of Concept

[▶ Watch: DNN watermarking: white-box vs. black-box approaches (4:00)]()

While the talk did not feature a live, interactive demonstration, the extensive experimental evaluation serves as a robust proof of concept for SSL-WM's design and capabilities. The methodology involved simulating real-world scenarios across diverse SSL models, datasets, and attack vectors.

Experimental Setup

• Self-supervised Learning Models: A representative set of SSL models was chosen for both CV and NLP:
• CV: SimCLR, MoCo V2, BYOL (contrastive-based, ResNet-18 encoder); BiGAN (generative-based, ResNet-18); CLIP (large-scale, ResNet-50 image encoder, Transformer text encoder).
• NLP: BERT (generative-based, Transformer architecture).
• Datasets: Eight benchmark datasets were used:
• CV: CIFAR-10, STL-10, CINIC-10, GTSRB.
• NLP: WikiText-103 (pre-training), SNLI, MRPC, IMDB (downstream).
• Pre-training and Watermark Embedding: SimCLR, MoCo V2, BYOL, and BiGAN were pre-trained on CIFAR-10. CLIP and BERT were officially released pre-trained models, fine-tuned to embed the SSL-WM watermark.
• Downstream Tasks (Adversary Simulation): To mimic an adversary stealing and customizing an encoder, watermarked encoders were transferred to various downstream tasks:
• CV Encoders: CIFAR-10 (best case), CINIC-10 (upper medium case), STL-10 (lower medium case), GTSRB (worst case).
• NLP Encoders (BERT): SNLI, MRPC, IMDB.

The transfer involved freezing the encoder and fine-tuning a classifier for CV, and fine-tuning both encoder and classifier for NLP, consistent with standard practices.

• Shadow Dataset (Ds):
• CV: Composed of 8,000 images from multiple domains (CIFAR-10, ImageNet, STL-10, GTSRB). The watermark pattern is depicted in Figure 4(a) in the original paper.
• NLP: Composed of randomly sampled instances from WikiText-103, SNLI, MRPC, IMDB. The watermark pattern used was the string 'bbb'.
• Platform: Experiments ran on Ubuntu 18.04 with two NVIDIA GeForce RTX 3090 GPUs (24GB memory each) and an Intel Xeon E5-2620 v4 CPU.

Evaluation and Results as Proof of Concept

The extensive evaluation results served as the "proof of concept" for SSL-WM:

• Effectiveness: Table III demonstrated successful ownership verification for all watermarked SSL models across diverse downstream tasks, with an average outlier index of 33.50 (min 3.67), comfortably exceeding the threshold of 3. Clean models never exceeded 1.22. T-SNE visualizations (Figure 2, Figure 6) visually confirmed that watermarked samples consistently formed a distinct, invariant cluster in the embedding space, validating the generalization mechanism.
• Fidelity and Efficiency: Watermark embedding incurred an average computational overhead of 1,991 minutes (22.81%) during training, deemed acceptable as a one-time process. Verification time was efficient, typically under 10 seconds for most models and up to 57.72 seconds for larger models like CLIP and BERT. Performance degradation on main tasks averaged only 0.41%.
• Robustness:
• Fine-tuning: Figure 3 showed that even after fine-tuning, MAD values remained significantly above 3 (e.g., SimCLR on GTSRB: 480.54), confirming robust verification.
• Pruning: Watermark remained detectable as long as main task performance was maintained (below 30-50% pruning rates). Beyond that, model utility collapsed, making it useless to the adversary.
• Input Preprocessing: Adding Gaussian noise (causing up to 3% accuracy drop) still resulted in high MAD values (e.g., 167.50 for SimCLR on CIFAR-10), confirming successful verification.
• Stealthiness: SSL-WM successfully evaded all tested detection methods:
• Neural Cleanse: Failed to reconstruct the watermark (Mask Jaccard Similarity near 0).
• ABS: Generated dissimilar triggers (average MJS 0.078).
• MNTD: Achieved 0% detection accuracy.
• Beatrix: True positive rate of only 0.2% to 0.4%.
• Februus: Could not effectively evade verification, despite impacting clean data accuracy.
• LOF: Average detection rate of only 1.38%.
• Comparison with SSLGuard: Direct comparison (Section V-E) showed SSLGuard consistently failed to verify ownership when transferred to downstream tasks (outlier index below 3), while SSL-WM successfully verified ownership with high confidence (e.g., 88.95 for SimCLR on CIFAR-10).

These comprehensive experimental results provide compelling evidence and a strong proof of concept for SSL-WM's ability to protect the intellectual property of SSL encoders in real-world, black-box scenarios.

Defensive Implications

[▶ Watch: Existing black-box watermarking for supervised learning models (4:30)]()

SSL-WM presents crucial insights and actionable strategies for both model owners and organizations utilizing self-supervised learning models.

For Original Model Owners:

• Proactive IP Protection: Owners of expensive, pre-trained SSL encoders should proactively embed SSL-WM during or immediately after the initial training phase. This one-time overhead (average 1,991 minutes or 22.81% of training time) is a worthwhile investment to safeguard significant IP.
• Leverage Multi-domain Shadow Datasets: When embedding the watermark, utilize a multi-domain shadow dataset (Ds) composed of samples from various task domains. This significantly enhances the watermark's generalization and transferability, ensuring it remains detectable even when the stolen encoder is applied to unforeseen downstream tasks.
• Robust Black-Box Verification: Adopt the proposed Shannon entropy and MAD-based outlier detection method for ownership verification. This statistical approach provides high confidence (at least 99.7% probability for outlier_index > 3) and is robust against variations in downstream task label counts.
• Maintain Stealthiness during Verification: To prevent adversaries from detecting and attempting to remove the watermark during verification, employ sporadic querying. By mixing clean and watermarked samples, with watermarked queries forming a small proportion, the overall feature representation of queries appears variant, thwarting detection attempts.
• Counter Ambiguity Attacks with Hashing: To prevent adversaries from forging an SSL-WM-style watermark and falsely claiming ownership, owners should constrain watermark generation. A hash function (wm = Hash(info, key)) can generate the trigger pattern, where info relates to the owner's identity and key is a secret. This makes it computationally infeasible for an adversary to reverse-engineer or generate a convincing forged pattern that the original watermarked encoder would recognize.
• Understand Pruning Trade-offs: Be aware that while SSL-WM is robust against moderate pruning (up to 30-50%), aggressive pruning (e.g., 90%) can remove the watermark. However, such extreme pruning also severely degrades the model's utility (e.g., CIFAR-10 accuracy dropping to 11.9%), rendering the stolen model practically useless to the adversary. This establishes a practical limit to the effectiveness of pruning attacks against SSL-WM.
• Hard Labels for Robustness: The use of hard labels (Shannon entropy calculated from class probabilities) for verification, rather than raw output confidence, provides resilience against confidence perturbation attacks. Attackers attempting to randomize classification results would need to introduce substantial noise, which would dramatically impact the model's primary task performance (e.g., reducing CIFAR-10 accuracy to 46.22%), making the model unusable.

For Organizations Acquiring or Deploying SSL Models:

• Awareness of IP Theft Risk: Recognize the significant economic incentive for IP theft of high-value SSL models. When acquiring pre-trained models from third parties, intellectual property verification is a critical due diligence step.
• Demand Proof of Ownership: If an SSL model is being licensed or acquired, organizations should demand proof of ownership. Techniques like SSL-WM provide a robust, black-box mechanism for original owners to furnish such proof.
• Ineffectiveness of Traditional Detection: Understand that conventional backdoor detection tools (e.g., Neural Cleanse, ABS, MNTD) designed for supervised learning backdoors are largely ineffective against advanced SSL watermarking schemes like SSL-WM (e.g., 0% detection accuracy for MNTD). New detection paradigms are needed if one suspects a model contains a hidden watermark.

By incorporating these defensive implications, the ecosystem surrounding self-supervised learning can better protect its valuable assets and mitigate the risks associated with intellectual property theft in this rapidly evolving domain.

Key Takeaways

• The significant investment in training Self-Supervised Learning (SSL) models makes them prime targets for intellectual property theft, necessitating robust ownership verification mechanisms.
• SSL-WM is a novel, generic, and completely black-box watermarking approach specifically designed to protect SSL encoders, addressing the critical challenges of unknown downstream tasks and black-box verification.
• The core technical innovation is training the watermarked encoder to map all watermarked inputs into an invariant representation space, enabling consistent classification by any subsequent downstream classifier.
• Ownership verification is achieved through Shannon entropy and Median Absolute Deviation (MAD) based outlier detection, providing a highly confident statistical method to distinguish watermarked models.
• SSL-WM demonstrates exceptional robustness against common adversarial attacks like fine-tuning, pruning, and input preprocessing, as long as the model's utility remains intact, and exhibits strong stealthiness, successfully evading state-of-the-art watermark detection methods.
• Experimental results confirm SSL-WM's superiority over prior SSL watermarking approaches (e.g., SSLGuard) in verifying ownership for encoders reused in downstream tasks, with minimal performance degradation and acceptable computational overheads.

About the Speaker(s)

Peizhuo Lv is the speaker for this talk. No further biographical details, such as title or company affiliation, were provided in the transcript or metadata.