Responsible Finetuning of Large Language Models

Responsible Finetuning of Large Language Models

Speakers: Ling Liu, Professor, Georgia Tech

Conference: MLSys 2025

YouTube: https://slideslive.com/39043020

Overview

This article delves into the critical and evolving challenges surrounding the responsible finetuning of Large Language Models (LLMs), with a particular emphasis on ensuring their safety and robustness in real-world applications. Presented by Professor Ling Liu of Georgia Tech at MLSys 2025, the talk underscores that while LLMs are becoming increasingly powerful and ubiquitous, their inherent limitations and risks, especially concerning safety, remain significant. The core message is that safety alignment, though crucial, is a brittle and complex process, prone to issues like catastrophic forgetting, contextual dependencies, and vulnerabilities introduced during downstream user finetuning.

Professor Liu's presentation moves beyond merely identifying problems, exploring novel mitigation strategies developed by her research group and others in the field. She introduces concepts like "vaccination" during safety alignment and "post-pruning" (Antidote) to enhance model resilience against harmful data and adversarial attacks. The talk also highlights the emerging threats from advanced prompt injection and multilingual vulnerabilities, advocating for a holistic approach that integrates guardrails throughout the entire LLM lifecycle, from training to inference. This work is paramount for anyone involved in developing, deploying, or researching LLMs, as it addresses the fundamental trustworthiness and reliability of these transformative AI systems, especially as they integrate into mission-critical domains and cross-modal applications.

Background

▶ Watch: Introduction and talk focus on safety alignment (0:00)

Large Language Models (LLMs) fundamentally operate as foundation models, initially pretrained on vast, often unlabeled, datasets with a focus on task-agnostic understanding. This pretraining establishes a broad base of knowledge and linguistic capabilities. However, to excel in specific tasks or adhere to particular guidelines, these foundation models undergo finetuning. This process tailors the model's behavior, often involving supervised learning on labeled data or reinforcement learning with human feedback (RLHF), to improve performance on benchmarks or align with desired output styles. Virtually all commercially released or open-source LLMs incorporate this two-stage process.

Despite their impressive capabilities, LLMs exhibit common limitations and inherent risks. These can be broadly categorized by their origin:

1. Data-related issues: Biases present in the training data, whether public or curated, can lead to skewed or unfair model outputs.
2. Training-related issues: The architectural choices and optimization processes during training, such as embedding representations, optimization algorithms, and attention mechanisms, can introduce approximations or errors. These might sometimes manifest as beneficial creativity but can also lead to factual misrepresentation or "bad" hallucinations.
3. Inference-related issues: Optimizations performed during the inference stage, such as KV caching and its compression techniques, can also contribute to unsafe factors or hallucinations.

One of the most persistent problems is hallucination, where models generate factually incorrect or nonsensical information with high confidence. While some forms of hallucination might be desirable for creative tasks, unchecked hallucination poses severe safety risks, particularly in critical applications. Professor Liu emphasizes that hallucination is an inherent property of these models, stating, "some hallucination may be good for creativity, but we have to learn how to mitigate that." The core challenge lies in understanding these multifactorial causes of unsafety and developing robust mitigation techniques, recognizing that a complete eradication of unsafe behaviors is likely impossible. This necessitates a continuous "arm race" against vulnerabilities, much like developing different vaccines for evolving viruses, rather than seeking a single panacea.

The conventional approach to addressing these safety concerns is safety alignment. This typically involves aligning the model's responses with human preferences, often collected as pairs of "bad" prompts and their desired "good" or "bad" answers. This data collection process is prohibitively expensive and labor-intensive, relying on human annotators to label preferences. Techniques like Reinforcement Learning from Human Feedback (RLHF) leverage these human preferences to continuously refine the model's policy and reward model, making it more conservative in responding to suspicious queries. However, this process is not without its own set of problems, including the immense cost of data, the subjective and contextual nature of "safety," and the risk of catastrophic forgetting, where the model's original capabilities or knowledge are degraded during the safety alignment process.

Key Findings

▶ Watch: Data, training, and inference: sources of unsafety (3:00)

Professor Ling Liu's talk highlights three critical, often interconnected, open issues that plague even well-aligned LLMs, challenging their presumed safety and robustness:

1. Catastrophic Forgetting due to Safety Alignment: Over-finetuning a model for safety can lead to a significant degradation of its performance on core tasks or its ability to recall previously learned information. This phenomenon, where enforcing safety constraints inadvertently erases or modifies beneficial knowledge, necessitates a delicate balance in the alignment process, which is inherently difficult to control. Data released by GPT itself illustrates how different finetuning stages for safety can result in varying levels of catastrophic forgetting.

1. Contextual and Size-Dependent Safety: The definition of "safe" is highly contextual, varying across countries, languages, cultural backgrounds, and specific tasks. What is considered private or acceptable in one region might not be in another. Furthermore, the talk reveals a counter-intuitive finding: larger, more powerful models, despite their superior reasoning capabilities, are often more vulnerable to certain types of attacks. Examples from GPT-3 and the Llama 3.1 family (specifically the 405 billion parameter version) show that while their reasoning ability for challenging datasets is excellent, they can be easily exploited to generate unsafe content, whereas smaller models sometimes exhibit greater inherent safety. Different finetuning techniques (e.g., Chain-of-Thought (COT), few-shot learning) also produce varied safety outcomes, complicating a universal approach.

1. Downstream Vulnerability from User Finetuning: Even a meticulously safety-aligned model, once released, can become unsafe if users perform further finetuning for downstream tasks. This vulnerability arises even when users employ benign data, if that data's latent representation (embedding) is "very close to the boundary of safe and unsafe." More alarmingly, very small amounts of harmful data (e.g., 5-10%, less than 20%) introduced during user finetuning can compromise the model's safety without significantly impacting its task accuracy. The underlying mechanism for this is embedding drift, where the hidden embeddings of the model shift in response to the new data, leading to unsafe behaviors.

To address these vulnerabilities, Professor Liu's group and others have developed novel mitigation techniques:

• Vaccination during Safety Alignment: This approach integrates a "vaccination" process during the initial safety alignment phase. By formulating the problem as a min-max optimization, the technique aims to make the model more resilient to potential harmful inputs or embedding drift during subsequent user finetuning. It achieves this by adaptively changing gradients and adding noise, which helps the model avoid drifting into unsafe regions without a significant accuracy penalty.

• Post-Pruning (Antidote): This technique, named Antidote, operates after both safety alignment and user finetuning. It involves a "one-shot pruning" of model parameters that are overly sensitive to harmful data. By identifying and removing these specific parameters, Antidote significantly improves the model's resilience to unsafe inputs while maintaining good accuracy. It's noted for its efficiency, being cheaper in terms of computational resources (clock time, GPU usage) than the vaccination method.

• Inference-time Mitigations for Advanced Prompt Injection: For attacks that bypass finetuning-based defenses, such as sophisticated adversarial prompt injection (including invisible and multilingual variants), inference-time techniques are crucial. These include:
• Detection-based methods: Initially using perplexity to flag abnormal prompts, though this proved vulnerable. Newer approaches focus on semantic filtering.
• Input Preprocessing: Powerful techniques like paraphrasing the input query multiple ways using an LLM and then ranking the responses, or using different tokenization schemes, have shown surprising effectiveness in mitigating prompt injection attacks.

A key overarching observation is the lack of comprehensive, open-source benchmarks for advanced safety challenges, which hinders systematic research and evaluation of mitigation strategies. The talk also points out that the integration of language with other modalities (e.g., vision in vision-language models) significantly aggravates safety problems, introducing new, complex challenges in judging and advising on safe behaviors.

Technical Deep Dive

▶ Watch: GPT-4 hallucination example with rephrased questions (4:30)

The technical core of Professor Liu's presentation revolves around understanding the mechanisms of LLM unsafety and proposing sophisticated algorithmic solutions.

Safety Alignment and its Pitfalls

Traditional safety alignment typically involves:

• Data Collection: Gathering pairs of prompts and human-preferred responses (safe vs. unsafe). This is an imitation-based data collection process.
• Supervised Learning / RLHF: Training the model using this data. Reinforcement Learning from Human Feedback (RLHF) is particularly common, where a reward model learns human preferences from ranked responses, and then a policy model is trained to maximize this reward.
• Optimization Goal: Iteratively training the model to assign low loss to safe answers and high loss to unsafe answers, making it more conservative.

However, this process is prone to catastrophic forgetting. When the model's weights are significantly altered to align with safety preferences, it can "forget" or degrade its performance on general tasks it was originally trained for. This is analogous to changing the fundamental logic of the model.

The Problem of Embedding Drift

A central technical contribution from Professor Liu's group is the identification of embedding drift as a key mechanism behind downstream vulnerability. When a well-aligned model is finetuned by a user, even with benign data, the hidden embeddings (internal representations) of the model can shift. If these new embeddings move too close to the "boundary of safe and unsafe" regions in the latent space, the model's behavior can become unsafe, even if its task-specific accuracy remains high. The research involved measuring the difference between hidden embeddings before and after finetuning with slightly harmful data, revealing significant drift. This drift, rather than direct accuracy loss, is the indicator of emerging unsafety.

Vaccination: A Min-Max Optimization Approach

To counteract embedding drift and enhance resilience during safety alignment, Professor Liu introduced a "vaccination" technique. This method frames the problem as a min-max optimization problem:

• Inner Optimization (Maximization): The goal is to identify the "worst-case" perturbation or noise that would maximize the alignment loss and cause the largest embedding drift if the model were to encounter harmful data. This is effectively learning how an attacker or harmful data could most effectively destabilize the model's safety.
• Outer Optimization (Minimization): Simultaneously, the model's gradients are adapted to minimize the impact of this worst-case perturbation. This makes the model inherently more resilient to such drifts. The process involves iteratively adding noise and adjusting the gradients to make the model robust to inputs that might push its embeddings towards unsafe regions. While the talk briefly mentions using Taylor matrices in this optimization, the core idea is a continuous adversarial training loop during the safety alignment phase. This leads to a model that is "vaccinated" against future harmful data or adversarial attacks by being trained to withstand such shifts.

Post-Pruning (Antidote): Targeted Parameter Removal

The Antidote technique offers an alternative, post-finetuning mitigation strategy. It is inspired by systems research on KV cache optimization and LoRA-like parameter efficient methods, which often involve ranking and pruning model parameters for efficiency. Antidote repurposes this concept for safety:

• Identification: After safety alignment and user finetuning, it identifies specific model parameters that are overly sensitive to harmful data. This is done through a "one-shot pruning" or filtering process.
• Removal: These identified parameters are then removed or pruned. The hypothesis is that these parameters, even if individually small, cumulatively contribute to the model's sensitivity to unsafe inputs. By removing them, the model becomes more resilient without significantly affecting its overall accuracy.
• Mechanism: Unlike vaccination, Antidote doesn't rely on continuous noise injection during training. It's a targeted, post-hoc adjustment to the model's architecture. The speaker notes that similar ideas are being explored in KV cache pruning for inference, where removing less important KV cache values can also reduce sensitivity to unsafe data.

Advanced Prompt Injection and Mitigation

The talk also details the technical nature of advanced adversarial prompt injection:

• Direct Instructions: Human-readable prompts that instruct the model to "forget all you have learned" or bypass safety protocols.
• Algorithm-Generated Injections: More insidious attacks where an algorithm generates subtle, often invisible or nonsensical, perturbations to the prompt. These are analogous to adversarial examples in computer vision, where minor pixel changes can fool a classification model without altering human perception.
• Multilingual Attacks: Leveraging the fact that LLMs are often primarily aligned for high-resource languages (e.g., English). Translating a harmful query into a low-resource language (e.g., Zulu), asking the model, and then translating the unsafe response back to English, can bypass English-centric safety filters.

Mitigation techniques for prompt injection are categorized:

• Detection-Based:
• Perplexity: Initially, detecting abnormal prompts by computing the perplexity of the token sequence. High perplexity might indicate an unusual, potentially malicious, input. However, this method proved susceptible to attacks.
• Semantic Filtering: A more robust approach that analyzes the semantic content of the prompt to identify malicious intent, rather than just statistical properties.
• Input Preprocessing:
• Paraphrasing: Using an LLM to rephrase the user's query into multiple variations. These variations are then fed to the model, and the responses are ranked for safety. This simple technique has shown remarkable effectiveness against adversarial injections.
• Re-tokenization: Similar to paraphrasing, but instead of rephrasing, the input sequence is re-tokenized using different tokenization schemes. This can disrupt the specific adversarial patterns that an attack relies on, leading to safer outputs.

These mitigation strategies represent a multi-pronged technical approach, acknowledging that no single defense is sufficient against the complex and evolving landscape of LLM safety threats.

Experimental Setup & Results

▶ Watch: Unsafety is inherent; mitigation and tailored solutions (6:00)

The presentation provides several examples and results to substantiate the discussed challenges and mitigation techniques, though specific, detailed experimental setups (e.g., full dataset names, exact hardware configurations, hyperparameter details) are not exhaustively covered in the transcript, focusing more on the qualitative and comparative outcomes.

Catastrophic Forgetting & Contextual Safety

• GPT-4 Data: The talk references internal data released by GPT (presumably OpenAI) demonstrating catastrophic forgetting. This data illustrates that different stages or methods of safety finetuning applied to GPT-4 resulted in varying levels of task performance degradation, directly supporting the claim that balancing safety and utility is difficult.
• Model Size vs. Safety: Experiments with GPT-3 and the Llama 3.1 family (including a 405 billion parameter version) showed that larger models, despite superior reasoning, were sometimes more vulnerable to unsafe outputs than smaller counterparts when faced with the same unsafe questions. This highlights the non-linear relationship between model scale, reasoning ability, and safety robustness.
• Finetuning Techniques: Studies on Llama 3.1 with different finetuning methodologies like Chain-of-Thought (COT) and few-shot learning showed varied safety scores and degraded quality on specific datasets after safety alignment, indicating that the choice of finetuning method significantly impacts safety outcomes.
• Adversarial Prompt Injection: Comparative results across three unspecified models indicated that for certain prompt injection attacks, powerful, big models were less resilient, becoming more vulnerable to exploitation. This reinforces the idea that increased capability can sometimes open new attack surfaces.

Downstream Vulnerability and Embedding Drift

• Embedding Drift Measurement: The research group conducted studies showing that finetuning a model with even a small percentage of harmful data (e.g., less than 20%, potentially 5-10%) or benign data close to the safety boundary caused significant embedding drift in the hidden layers of the model. This drift, measured as the difference in hidden embeddings before and after finetuning, correlated strongly with the model exhibiting unsafe behaviors, even if standard accuracy metrics remained unaffected.

Vaccination Technique

• Resilience and Accuracy: The "vaccination" approach was evaluated on its ability to make safety alignment more resilient to user-level low-quality data during downstream finetuning.
• Results: The technique demonstrated significantly improved resilience (lower harmful score, indicating fewer unsafe responses) compared to baseline safety alignment.
• Accuracy Penalty: While the approach introduced some minor accuracy penalty, it was shown to keep the penalty "very close" to original cases, suggesting an effective balance.
• Generality: The vaccination method was tested across "different models" and "different datasets," showing consistent positive effects.
• Embedding Drift Reduction: Crucially, measurements confirmed that vaccination substantially reduced the embedding drift caused by harmful data, though it did not completely eliminate it. This directly supports the proposed mechanism of action.

Post-Pruning (Antidote)

• Harmful Score and Accuracy: The Antidote technique was compared against vaccination and other existing approaches.
• Results: Antidote achieved a "much better" harmful score (higher resilience to harmful data) than vaccination and other methods. Simultaneously, it maintained "pretty good" finetuning accuracy.
• Cost-Effectiveness: A significant finding was Antidote's efficiency. It was reported to be "cheaper" in terms of "clock time and GPU usage" compared to the vaccination method, which demands more resources due to its iterative noise injection and gradient adaptation.

Inference-time Mitigation

• Paraphrasing & Re-tokenization: Research on input preprocessing techniques showed that simply paraphrasing a query using another LLM or applying different tokenization schemes could "improve the result very easily" against advanced prompt injection attacks. These studies reportedly crafted small datasets to demonstrate the effectiveness of these simple yet powerful methods, outperforming direct detection using perplexity.

Overall, the experimental results underscore the severity of LLM safety challenges and validate the effectiveness of the proposed "vaccination" and "Antidote" techniques, particularly in their ability to enhance resilience and reduce embedding drift while managing accuracy tradeoffs. The cost-effectiveness of Antidote positions it as a practical solution for deployment.

Practical Implications

▶ Watch: Objective of safety alignment: prevent unsafe answers (7:30)

The insights from Professor Liu's talk carry profound practical implications for a wide range of stakeholders in the AI/ML ecosystem, from model builders to infrastructure teams and deployers.

For Model Builders and Developers:

• Rethink Safety Alignment: Current safety alignment methods are demonstrably brittle and not robust enough. Model developers must move beyond simple RLHF and consider integrating more sophisticated techniques like "vaccination" during initial alignment to build in resilience from the start.
• Beware Downstream Finetuning: Releasing a "safe" foundation model does not guarantee its safety after users perform further finetuning. Developers need to educate users about the risks of embedding drift and potentially provide tools or guidelines for safer downstream adaptation, or even integrate mechanisms like Antidote into their model releases.
• Contextual Safety is Key: A "one-size-fits-all" approach to safety is insufficient. Model developers must consider the diverse contexts (cultural, linguistic, task-specific) in which their models will be deployed. This might necessitate context-aware safety filters or region-specific alignment strategies.
• Larger Models, More Vulnerability: The finding that larger, more reasoning-capable models can be more vulnerable to certain attacks is a critical warning. Scaling up models does not automatically equate to increased safety; in fact, it may introduce new, harder-to-detect vulnerabilities. Developers need to rigorously test large models for these specific failure modes.

For Infrastructure Teams and Deployers:

• Guardrails Everywhere: The call for "guardrails everywhere" in the LLM workflow (input, training, finetuning, output) is a crucial architectural principle. Infrastructure teams should implement multi-layered safety mechanisms, not just rely on post-output filtering.
• Inference-Time Defenses: Given the prevalence of advanced adversarial prompt injection (including invisible and multilingual attacks), robust inference-time defenses are non-negotiable. Deployers should integrate input preprocessing techniques (e.g., paraphrasing, re-tokenization) and advanced semantic detection mechanisms to catch and neutralize malicious prompts before they reach the core model.
• Resource Tradeoffs: Techniques like "vaccination" can be computationally more demanding (GPU usage, clock time) than traditional alignment. Infrastructure teams must factor these increased resource requirements into their planning and budget. Conversely, methods like Antidote offer a more cost-effective post-hoc solution.
• Monitoring and Feedback Loops: Continuous monitoring of model outputs for unsafe content, coupled with robust feedback loops, is essential. The "arm race" against attackers requires constant vigilance and adaptation of defenses.

For Practitioners and Researchers:

• Lack of Benchmarks: The absence of comprehensive, open-source benchmarks for advanced safety challenges is a major limitation. Researchers and industry consortia need to prioritize the development and sharing of such benchmarks to accelerate progress in defensive AI.
• Cross-Modal Challenges: As AI moves towards cross-modal and vision-language models, safety problems are exacerbated. Practitioners in these emerging areas must anticipate and proactively address the magnified complexity of ensuring safety across multiple modalities.
• Balancing Act: There's an inherent tension between model capability/innovation and safety. While safety is paramount, especially in critical domains, over-constraining models can stifle creativity or performance in areas like scientific discovery. Finding the right balance will require careful domain-specific analysis and transparent tradeoff decisions.
• Hardware Opportunities: The discussion on parameter pruning (like Antidote) and its analogy to LoRA and KV cache optimization suggests new avenues for hardware-software co-design. Hardware architects could explore specialized accelerators or memory structures that efficiently support safety-specific pruning or adversarial training workloads.

Limitations and Future Directions:

Despite the progress, the talk acknowledges that no solution is a "panacea." The "arm race" will continue, requiring ongoing research into more robust perturbation techniques, better understanding of embedding dynamics, and more sophisticated inference-time defenses. The contextual nature of safety remains a profound challenge, demanding more adaptive and personalized safety models. The sheer cost and difficulty of collecting human preference data for safety alignment continue to be a bottleneck, hinting at the need for more automated or self-supervised safety learning methods.

In essence, the talk serves as a critical call to action: LLM safety is not a solved problem but a dynamic and complex frontier requiring continuous innovation, interdisciplinary collaboration, and a holistic approach to design, deployment, and ongoing operation.

Key Takeaways

• LLM Safety is Inherently Brittle: Despite sophisticated safety alignment, Large Language Models remain vulnerable to various forms of unsafety, including catastrophic forgetting, contextual misinterpretations, and downstream compromises.
• Downstream Finetuning Poses Significant Risks: Even well-aligned models can become unsafe when users perform further finetuning, even with benign data, due to subtle "embedding drift."
• Larger Models are Not Necessarily Safer: Counter-intuitively, more powerful and reasoning-capable models (e.g., Llama 3.1 405B) can exhibit increased vulnerability to certain adversarial attacks compared to smaller models.
• Novel Mitigation Strategies Show Promise: Techniques like "vaccination" (min-max optimization during alignment to reduce embedding drift) and "Antidote" (post-finetuning parameter pruning) offer effective ways to enhance model resilience against harmful data and adversarial inputs.
• Multi-Layered Defenses are Essential: Robust defense against advanced adversarial prompt injection (including invisible and multilingual attacks) requires a combination of inference-time techniques such as input paraphrasing, re-tokenization, and semantic detection, alongside improved finetuning.
• Holistic Guardrails and Benchmarks are Crucial: A truly safe LLM ecosystem requires guardrails at every stage of the workflow (input, training, finetuning, output) and a significant investment in open, comprehensive benchmarks to drive and evaluate research progress.

About the Speaker(s)

Professor Ling Liu is a distinguished Professor at Georgia Tech. Her research focuses on systems-related aspects of AI and ML, with a particular interest in the safety and robustness of large language models. The work presented in this talk reflects the collaborative efforts of her current and recently graduated students, who have contributed to various aspects of her research, including the development of advanced safety alignment and mitigation techniques. Her insights bridge the gap between theoretical understanding of LLM vulnerabilities and practical solutions for building more responsible AI systems.