Model Immunization from a Condition Number Perspective

Model Immunization from a Condition Number Perspective

Speakers: Amber Yijia Zheng, Cedar Site Bai, Brian Bullins, Raymond A. Yeh

Conference: ICML 2025

YouTube: https://slideslive.com/39044088

Overview

In an era defined by the rapid advancement and widespread accessibility of generative AI, the responsible deployment of powerful open-weight models like Stable Diffusion and DeepFloyd presents a significant challenge. While these models democratize content creation, their inherent flexibility also introduces the risk of malicious adaptation—the intentional fine-tuning of a model to generate harmful, unethical, or copyrighted content. This talk, presented by Cedar Site Bai from Purdue University, along with collaborators Amber Yijia Zheng, Brian Bullins, and Raymond A. Yeh, introduces a novel framework for model immunization designed to counteract this threat.

The core idea behind model immunization is to modify the weights of a released model such that it becomes inherently difficult to adapt it for harmful purposes, while retaining its utility for benign tasks. The presented work approaches this problem from a unique perspective, leveraging the concept of the condition number of the Hessian matrix to quantify the difficulty of fine-tuning. By strategically manipulating this condition number during the pre-training phase, the researchers aim to make malicious adaptation computationally prohibitive, thereby fostering safer and more responsible AI deployment.

This research is particularly relevant to the ongoing discourse surrounding AI safety, ethics, and intellectual property. As open-source AI models become foundational tools across industries, ensuring their robustness against misuse is paramount. The proposed condition number-based immunization framework offers a principled, mathematically grounded approach to achieving this, moving beyond reactive content filtering to proactive model design that builds resilience against malicious intent directly into the model's architecture.

Background

▶ Watch: Threat of malicious adaptation on open-weight models. (0:38)

The proliferation of open-weight models has revolutionized AI development, making advanced capabilities accessible to a broad audience. However, this accessibility comes with a critical vulnerability: the ease with which these models can be adapted, or fine-tuned, for unintended or malicious purposes. While initial pre-training datasets are often curated to exclude undesirable content (e.g., due to safety, ethical, or copyright concerns), the very mechanism that makes these models adaptable—transfer learning—also allows harmful concepts to be re-injected through subsequent fine-tuning. The challenge lies in developing a defense mechanism that modifies the released model weights to resist such malicious adaptation.

The talk frames this problem within the context of transfer learning, where a model is typically decomposed into a feature extractor (f_theta) and a task-specific linear adapter (g_omega or h_w). The feature extractor f_theta is initially pre-trained on a large dataset (D_P) along with an adapter g_omega. For a harmful task (D_H), this g_omega is replaced with h_w, and the model is fine-tuned. The researchers specifically focus on the scenario where a "bad actor" performs linear probing, updating only the adapter weights w while keeping the feature extractor f_theta fixed. This simplification allows for an analytical characterization of the fine-tuning process, particularly through the lens of an L2 regression loss.

A central concept introduced to quantify the "difficulty" of fine-tuning is the condition number. In numerical analysis and optimization, the condition number of a matrix (specifically, the Hessian matrix in optimization) is defined as the ratio of its largest singular value to its smallest singular value. A high condition number indicates that the optimization problem is "ill-conditioned," meaning that small changes in input can lead to large changes in the output, and gradient-based optimization methods (like steepest descent) will converge very slowly. Conversely, a low condition number indicates a "well-conditioned" problem, leading to faster convergence. Therefore, the core hypothesis is that increasing the condition number of the Hessian for a harmful fine-tuning task will make that task significantly harder (slower to converge), while maintaining a low condition number for benign tasks will preserve model utility. This approach shifts the defense mechanism from detecting harmful outputs to preventing their learning in the first place.

Key Findings

▶ Watch: Feasibility of immunization based on singular vector alignment. (2:58)

The research makes several pivotal findings and contributions towards principled model immunization:

1. Feasibility of Immunization: The talk demonstrates analytically that model immunization is indeed possible. By characterizing the Hessian matrix as a quadratic form involving the feature extractor theta and the data covariance matrix K (Hessian = theta^T K theta), the singular values of the Hessian are shown to depend on both theta and K. The feasibility of making one task ill-conditioned while keeping another well-conditioned hinges on the alignment (or relative angle) between the singular vectors of the data covariance matrices for the two tasks. Only in the extreme, perfectly aligned case is immunization impossible.

1. Condition Number as an Immunization Metric: The work establishes the condition number as a precise and quantifiable metric for assessing the difficulty of fine-tuning. A successful immunization strategy is defined by three criteria: (1) fine-tuning on harmful tasks should be significantly harder (larger condition number) than on raw input, (2) fine-tuning on other, benign tasks (including the pre-training task) should not be more difficult, and (3) the model's performance on the pre-training task must be preserved.

1. Novel Regularizers for Condition Number Control: To achieve the desired manipulation of condition numbers, the researchers propose a novel objective function incorporating two differentiable regularizers:

• R_ill (Ill-Conditioning Term): Applied to the harmful Hessian, this term is the inverse of the gap between the average singular value and the smallest singular value (related to the Frobenius norm). Minimizing R_ill actively increases the condition number, making fine-tuning harder. It possesses desirable properties: it upper bounds 1/log(kappa) (where kappa is the condition number), is differentiable with a closed-form gradient, and monotonically increases the condition number when updated with gradient steps.
• R_well (Well-Conditioning Term): Adapted from Nan et al., this regularizer is applied to the pre-training task's Hessian. It measures the difference between the largest singular value and the average singular value, thereby maintaining a low condition number and ensuring efficient learning for benign tasks.

1. Practical Gradient-Based Optimization Algorithm: Based on these regularizers, a practical gradient-based algorithm is proposed. The adapter weights are updated using gradients solely from the pre-training loss, while the feature extractor f_theta is updated using gradients from all three terms: the pre-training loss, R_well, and R_ill. This selective update strategy allows for targeted manipulation of the feature extractor's properties.

1. Relative Immunization Ratio (RIR): A new metric, the Relative Immunization Ratio (RIR), is introduced to quantify the overall immunization effectiveness. RIR is defined as the ratio of the ill-conditioning achieved on the harmful task (numerator, larger is better) to the degradation on the pre-training task (denominator, smaller is better). A higher RIR signifies better immunization.

1. Empirical Validation Across Diverse Models: The framework's effectiveness is rigorously validated across various model types and tasks:

• Linear Regression: On the House Price dataset, the method achieved the highest RIR, demonstrating faster convergence on the pre-training task and significantly slower convergence on the harmful task compared to baselines.
• Binary Classification: On MNIST, the method successfully immunized across all 90 possible digit pairs, indicating broad applicability.
• Deep Models: When applied to ResNet-18 and Vision Transformer models initialized with ImageNet pre-trained weights, the method yielded the highest RIR on harmful datasets like Stanford Cars and Country211. Crucially, it maintained competitive ImageNet classification accuracy (pre-training performance) while achieving the lowest test accuracy on the harmful datasets, directly confirming the link between higher RIR and stronger immunization.

These findings collectively present a robust and mathematically sound approach to model immunization, offering a promising path towards building more secure and responsible open-source AI models.

Technical Deep Dive

▶ Watch: Gradient-based algorithm for updating feature extractor. (4:20)

The core of this immunization framework lies in its ability to precisely control the condition number of the Hessian matrix associated with different fine-tuning tasks. For a gradient-based optimization algorithm like steepest descent, the convergence rate is inversely proportional to the condition number for strongly convex functions. A larger condition number implies a "flatter" or more elongated loss landscape along some directions and steeper along others, making it difficult for gradient descent to find the minimum efficiently, thus leading to slower convergence.

The problem is formulated in a transfer learning setting where a pre-trained feature extractor f_theta extracts representations, and a linear adapter w (or h_w for harmful tasks) performs the final prediction. The bad actor is assumed to perform linear probing, updating only w. The loss function considered is an L2 regression loss, which allows for analytical characterization. The Hessian matrix for this linear adapter w can be expressed as Hessian = theta^T K theta, where theta represents the feature extractor and K is the data covariance matrix for the specific task. The singular values of this Hessian, which dictate the condition number, are therefore directly influenced by both the feature extractor theta and the statistical properties (K) of the data for a given task.

The feasibility of immunization, i.e., making one task ill-conditioned while keeping another well-conditioned, depends on the alignment of the singular vectors of the data covariance matrices (K_P for pre-training and K_H for harmful) in the feature space. If these singular vectors are perfectly aligned, then manipulating theta to affect one task will inevitably affect the other similarly, making immunization impossible. However, in most practical scenarios, such perfect alignment is rare, allowing for differential manipulation.

To implement this control, the researchers propose an objective function that combines the original pre-training loss with two novel regularizers:

1. Ill-Conditioning Regularizer (R_ill): This term is designed to increase the condition number of the Hessian for the harmful task. It is defined as the inverse of the gap between the average singular value and the smallest singular value of the harmful Hessian. Mathematically, if sigma_i are the singular values of the Hessian H_H, and sigma_min is the smallest singular value, then R_ill is related to 1 / ( (sum(sigma_i) / rank(H_H)) - sigma_min ). Minimizing this term forces sigma_min to be much smaller relative to the average, thus increasing the condition number kappa = sigma_max / sigma_min. The talk specifies that R_ill "upper bounds one over log of the condition number kappa," meaning that minimizing R_ill will effectively maximize log(kappa) and thus kappa. Crucially, R_ill is designed to be differentiable with a closed-form gradient, enabling its integration into standard gradient-based optimization. Furthermore, updates using R_ill are guaranteed to monotonically increase the condition number, providing a reliable mechanism for control.

1. Well-Conditioning Regularizer (R_well): This term, adapted from Nan et al., aims to maintain a low condition number for the pre-training task's Hessian, ensuring that the model remains efficient for benign tasks. It measures the difference between the largest singular value (sigma_max) and the average singular value of the pre-training Hessian. By minimizing this difference, the singular values are encouraged to be clustered closely, resulting in a low condition number. The speaker notes that the "spectral norm is just the max singular value and P is the dimension of the matrix," implying R_well might be related to sigma_max - (sum(sigma_i) / P).

The overall training algorithm proceeds as follows:

• The adapter weights (g_omega for pre-training, or h_w for harmful tasks during evaluation) are updated using gradients derived solely from the pre-training loss. This ensures that the adapter learns the benign task effectively.
• The feature extractor weights (f_theta) are updated using gradients from a composite loss function that includes all three terms: the pre-training loss, R_well (to maintain well-conditioning for benign tasks), and R_ill (to induce ill-conditioning for harmful tasks). This allows the feature extractor to learn representations that inherently make harmful fine-tuning difficult.

The Relative Immunization Ratio (RIR) serves as the key evaluation metric. It quantifies the degree of ill-conditioning achieved on the harmful task relative to any degradation in well-conditioning on the pre-training task. A higher RIR indicates a more successful immunization, balancing the objectives of making harmful tasks difficult while preserving benign task performance. The mathematical formulation ensures that the denominator of RIR captures the "degradation on the pre-training task," implying a measure of how much harder the pre-training task has become.

Experimental Setup & Results

▶ Watch: Introducing Relative Immunization Ratio (RIR) metric. (5:00)

The effectiveness of the proposed condition number-based immunization framework was evaluated across a range of tasks and model architectures, employing various datasets and baselines.

1. Regression Task (House Price Dataset):

• Setup: The initial experiments utilized a linear regression task on a house price dataset. This choice was strategic because the ground truth optimal solution can be analytically obtained, allowing for precise measurement of the speed of convergence by tracking the distance between the current solution and the optimum.
• Baselines:
• R_ill only: A variant that only applies the ill-conditioning regularizer without R_well to maintain well-conditioning for the pre-training task.
• EMA (Bilevel Optimization): A bilevel optimization framework proposed by Jun and Yeh, representing a state-of-the-art approach to related problems.
• Opt-kappa (Naive Variant): A straightforward approach that directly appends the condition numbers to the pre-training loss, serving as a naive baseline for comparison.
• Results: The proposed method achieved the highest Relative Immunization Ratio (RIR). This translated directly into fast convergence on the pre-training task and significantly slower convergence on the harmful task. Convergence curves visually demonstrated this: for the harmful task, the proposed method's curve lay significantly above the identity baseline (indicating slower convergence), while for the pre-training task, it lay below (faster convergence). Only Opt-kappa and the proposed method met both criteria, with the proposed method achieving even slower convergence on the harmful task, indicating stronger immunization.

2. Binary Classification (MNIST Dataset):

• Setup: The framework was extended to a binary classification task using the MNIST dataset. In this setting, one digit was designated as the pre-training task (e.g., classifying '0' vs. 'not 0'), and another digit was chosen as the harmful task (e.g., classifying '1' vs. 'not 1').
• Evaluation: Results were visualized using a grid cell where each cell corresponded to a unique digit pair used for pre-training and harmful tasks. The number inside each cell represented the log RIR. Blue cells indicated successful immunization (higher RIR), while red cells indicated failure (either pre-training became harder or harmful task became easier).
• Results: The proposed method demonstrated remarkable robustness, succeeding across all 90 possible digit pairs. This broad success highlights the generalizability of the condition number-based approach to different classification problems.

3. Deep Models (ResNet-18 and Vision Transformer):

• Setup: To validate the framework's efficacy on more complex, real-world models, experiments were conducted using ResNet-18 and Vision Transformer architectures. The feature extractors were initialized with models pre-trained on ImageNet. Instead of comparing against an identity matrix, the RIR was computed against the initial pre-trained feature extractor (theta_0). The adapter Hessian was calculated using the covariance of the extracted features (x_tilde).
• Harmful Datasets: Stanford Cars and Country211 were chosen as the harmful datasets, representing distinct fine-tuning challenges.
• Metrics: Beyond RIR, the ImageNet classification accuracy (representing pre-training task performance) and the test accuracy on the harmful dataset were reported. A lower test accuracy on the harmful dataset would directly verify stronger immunization.
• Results: The proposed method consistently yielded the highest Relative Immunization Ratio across these deep models and harmful datasets. Critically, it maintained competitive performance on the pre-training task (ImageNet classification accuracy), demonstrating that immunization did not significantly degrade the model's primary utility. Furthermore, the method consistently achieved the lowest test accuracy on the harmful datasets, directly confirming that a higher RIR indeed corresponds to stronger, more effective immunization against malicious adaptation.

These comprehensive experimental results across diverse tasks and model complexities strongly validate the effectiveness and robustness of the condition number-based framework for model immunization.

Practical Implications

▶ Watch: Classification results on MNIST and deep models. (6:20)

The model immunization framework presented in this talk carries significant practical implications for various stakeholders involved in the AI ecosystem, from model builders to deployment teams.

For model builders and researchers, this work provides a principled, mathematically grounded methodology to design and pre-train open-weight models that are inherently more resilient to misuse. Instead of relying solely on post-hoc filtering or reactive measures, builders can proactively embed defense mechanisms into the model's learning process. This could become a standard component of responsible AI development pipelines, where models are not just evaluated for performance but also for their "immunization" against harmful adaptations. The specific regularizers (R_ill and R_well) and the gradient-based algorithm offer concrete tools for implementation.

For infrastructure teams and model deployers, this framework offers a path towards releasing safer open-source models with greater confidence. The ability to quantify immunization strength via the Relative Immunization Ratio (RIR) provides a metric that can be incorporated into model release criteria. Deploying immunized models could reduce the operational burden of monitoring and mitigating malicious adaptations after deployment, shifting the focus towards prevention.

The concept of manipulating condition numbers to control fine-tuning difficulty also opens up new avenues for transfer learning research. It suggests that the "transferability" of features might not just be about positive knowledge transfer but also about strategically hindering negative transfer or adaptation to undesirable tasks.

However, there are important tradeoffs and limitations to consider:

1. Computational Cost: Incorporating condition number-based regularizers might add computational overhead during the pre-training phase, especially for very large models or complex Hessians. While the proposed regularizers have closed-form gradients, the singular value decomposition (SVD) operations required to compute them can be expensive.
2. Optimizer Dependence: A crucial point raised during the Q&A was the dependence of condition number sensitivity on the optimizer used. While the talk states that the framework was tested with Adam on deep models, the primary analytical motivation relies on the convergence properties of SGD or steepest descent, which are highly sensitive to ill-conditioning. Newer optimizers like K-FAC or Muon are designed to be more robust to ill-conditioned data. The effectiveness of this immunization strategy against such advanced optimizers remains an open question and an area for future research.
3. Scope of "Malicious Adaptation": The current framework focuses on linear probing scenarios where only a linear adapter is fine-tuned. While this is a common and powerful form of transfer learning, malicious actors might fine-tune the entire feature extractor f_theta or employ more sophisticated non-linear adaptation techniques. The generalizability of the condition number approach to these more complex adaptation scenarios needs further investigation.
4. Definition of "Harmful": The experiments primarily use classification/regression tasks (e.g., MNIST digits, specific datasets). Applying this to the broader and more nuanced problem of "harmful content generation" in large generative models (e.g., generating deepfakes, hate speech) would require careful definition of "harmful tasks" and how they translate to fine-tuning objectives.
5. Performance Preservation: While the results show "competitive performance" on pre-training tasks, there might be a subtle degradation or a limit to how much immunization can be achieved without impacting desired model capabilities. The RIR helps balance this, but the exact sweet spot might vary by application.

Despite these considerations, this work represents a significant step towards developing a principled, robust, and proactive defense against the misuse of open-weight AI models, contributing substantially to the field of responsible AI.

Key Takeaways

• Model immunization is a crucial strategy to defend open-weight AI models against malicious adaptation, making it difficult to fine-tune them for harmful purposes.
• The condition number of the Hessian matrix serves as a quantifiable metric for fine-tuning difficulty; increasing it for harmful tasks slows down convergence.
• A novel objective function is proposed, incorporating R_ill (to increase harmful task condition number) and R_well (to maintain pre-training task well-conditioning), alongside the original pre-training loss.
• A gradient-based algorithm effectively updates the feature extractor using these regularizers, demonstrating that the approach is practical for both linear and deep models.
• The Relative Immunization Ratio (RIR) is introduced as a key metric, showing superior immunization by achieving higher RIRs across regression, classification (MNIST), and deep learning tasks (ResNet-18, Vision Transformer).
• The framework successfully preserves pre-training task performance (e.g., ImageNet accuracy) while significantly reducing the model's ability to learn harmful tasks, making fine-tuning for malicious purposes much harder.

About the Speaker(s)

The primary presenter of this work was Cedar Site Bai, affiliated with the Department of Computer Science at Purdue University. This research is a collaborative effort, undertaken as joint work with Amber Yijia Zheng, Brian Bullins, and Raymond A. Yeh. While specific titles or affiliations for all co-authors beyond Purdue University were not detailed in the transcript, their collective expertise has contributed to this innovative approach to model immunization.