Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models

Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models

Speakers: Anshuman Chhabra, Assistant Professor, University of South Florida; Bo Li; Jian Chen; Prasant Mohapatra; Hongfu Liu

Conference: ICML 2025

YouTube: https://slideslive.com/39044010

Overview

In the rapidly evolving landscape of deep learning, the quality and relevance of training data are paramount. Models, especially large language models (LLMs), often learn undesirable behaviors or exhibit suboptimal performance due to noisy, mislabeled, or otherwise detrimental samples within their vast training datasets. The talk "Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models," presented by Anshuman Chhabra of the University of South Florida, introduces a novel and computationally efficient approach to address this critical challenge. This work, co-authored with Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu, proposes Outlier Gradient Analysis (OGA) as a method to pinpoint and remove these problematic samples, thereby enhancing model performance and interpretability.

The core problem OGA tackles is data valuation – understanding the contribution of individual training samples to a model's final utility. While traditional methods like leave-one-out retraining are prohibitively expensive, and established approximations like influence functions still grapple with computational bottlenecks, OGA offers a pragmatic alternative. By focusing on the distinctive characteristics of gradients associated with detrimental samples, the research posits that these samples can be identified as statistical outliers in the gradient space. This approach not only sidesteps the computational intensity of second-order methods but also provides a powerful tool for tasks ranging from model debugging and robust training to fairer data compensation schemes, marking a significant step forward in making data-centric AI more accessible and effective.

Background

▶ Watch: Introduction to Outlier Gradient Analysis and Data Valuation (0:00)

The pervasive challenge of data quality in machine learning underpins the need for effective data valuation. At its heart, the data valuation problem seeks to answer a fundamental question: can we curate our training data to include only samples that foster desirable model outcomes, while excluding those that introduce noise or lead to incorrect behaviors? This has broad implications, from improving classification accuracy by removing mislabeled samples to debugging complex LLMs that exhibit unexpected or erroneous responses. Furthermore, in future scenarios, efficient data valuation could enable fair compensation for users contributing high-quality data, distinguishing them from those providing less valuable or even harmful inputs.

Historically, the most straightforward approach to data valuation is leave-one-out retraining. This method involves training a model, removing a single data point, retraining the model, and then measuring the change in performance. While conceptually simple, its computational cost is astronomical, making it intractable for any non-trivial dataset or model architecture. For instance, retraining a large language model with millions or billions of parameters for each of hundreds of thousands or millions of training samples is practically impossible.

To circumvent this intractability, researchers turned to influence functions. Derived from robust statistics, influence functions approximate the effect of removing a training sample without requiring full retraining. The general form of an influence function relies on three key components:

1. The first-order gradient of the impact function (the metric being measured, e.g., loss or accuracy) with respect to the model parameters.
2. The inverse of the Hessian matrix of the loss function.
3. The gradient of the loss function with respect to the model parameters for the specific training sample whose influence is being calculated.

While influence functions offer a theoretical pathway, their practical application has been severely limited by the computational expense of inverting the Hessian matrix. Exact Hessian inversion scales cubically with the number of parameters, rendering it infeasible for deep learning models with millions or billions of parameters. Various approximations, such as LiSSA (Linear Subsampled Stochastic Average Approximation), have been proposed to estimate the Hessian inverse-vector product, but these still represent a significant computational bottleneck.

A simpler, first-order approximation is TracIn (Tracing Gradient with the Identity Hessian Approximation), introduced by Pruthi et al. in 2019. TracIn simplifies the influence function by assuming the Hessian matrix is an identity matrix. This reduces the calculation to a simple inner product between the gradient of the impact function and the gradient of the loss function for the sample in question. Essentially, TracIn measures the similarity between these gradient terms in the vector space. While significantly more efficient than Hessian-based methods, TracIn's simplistic assumption often leads to unsatisfactory performance, as it may not capture the complex interactions that truly define a sample's influence. This gap highlights the need for a method that is both computationally efficient and sufficiently accurate to identify detrimental samples effectively.

Key Findings

▶ Watch: Influence Functions: Approximating Leave-One-Out Retraining (3:05)

The research presented in this talk hinges on two critical observations specific to the problem of identifying detrimental training samples:

1. Predominance of Positive Contributions: In the context of empirical risk minimization, the vast majority of training samples contribute positively to the model's overall utility and performance. This implies that truly "detrimental" samples are likely to be a minority within the dataset.
2. Sample-Specific Gradient Dominance: For any given training sample, its influence information is primarily driven by the last gradient term in the influence function formulation—specifically, the gradient of the loss function with respect to the model parameters for that particular sample. Other terms, like the impact function gradient and the Hessian, are either constant or shared across samples, making the sample-specific loss gradient the primary differentiator of influence.

Building on these observations, the authors formulated a powerful hypothesis: detrimental samples are not only few in number but also possess gradients that are out-of-distribution (OOD) compared to the gradients of beneficial samples. In essence, if a sample is causing issues, its contribution to the model's learning (as reflected in its gradient) will deviate significantly from the norm established by useful samples.

This hypothesis was empirically verified using synthetic datasets. For a linearly separable dataset with label-flipped noisy samples and a non-linearly separable half-moons dataset also corrupted with noise, the gradients of the noisy, detrimental samples consistently appeared as clear outliers in the gradient space. This visual and empirical confirmation provided the foundation for the proposed solution.

The core contribution, Outlier Gradient Analysis (OGA), directly leverages this finding. Instead of grappling with the computationally intensive Hessian, OGA proposes a simpler yet effective strategy:

1. Compute the gradients for all training samples.
2. Apply an outlier analysis algorithm to this gradient space.
3. Identify and remove the samples whose gradients are detected as outliers.
4. Retrain the model on the pruned dataset.

The results demonstrated that OGA significantly improves model performance across various scenarios. It achieved top performance in noisy learning regimes with deeper models like ResNet-34 and ResNet-18 on noisy CIFAR datasets (featuring human annotator noise). It also showed superlative performance on GLUE datasets with synthetic label flipping using a RoBERTa model. Furthermore, OGA proved effective in identifying influential samples for LLMs on specific benchmarks, achieving near-perfect performance in these controlled settings. These findings collectively establish OGA as an efficient and competitive method for identifying detrimental training samples across diverse deep learning architectures and tasks.

Technical Deep Dive

▶ Watch: The Hessian Bottleneck in Influence Function Computation (4:50)

The technical foundation of Outlier Gradient Analysis (OGA) is rooted in re-evaluating the components of traditional influence functions and identifying a more efficient pathway to detect detrimental samples. As established, the influence function for a training sample $x_j$ on a test sample $x_{test}$ can be approximated as:

$Influence(x_j, x_{test}) \approx \nabla_{\theta} L(x_{test}, \hat{\theta})^\top H_{\hat{\theta}}^{-1} \nabla_{\theta} L(x_j, \hat{\theta})$

Where:

• $L$ is the loss function.
• $\hat{\theta}$ are the model parameters after training.
• $\nabla_{\theta} L(x_{test}, \hat{\theta})$ is the gradient of the loss on the test sample with respect to the model parameters (often referred to as the impact function gradient).
• $H_{\hat{\theta}}^{-1}$ is the inverse of the Hessian matrix of the loss function with respect to the model parameters.
• $\nabla_{\theta} L(x_j, \hat{\theta})$ is the gradient of the loss on the training sample $x_j$ with respect to the model parameters.

The critical bottleneck, as previously discussed, is the computation and inversion of the Hessian matrix $H_{\hat{\theta}}$. OGA sidesteps this by focusing on the observation that the third term, $\nabla_{\theta} L(x_j, \hat{\theta})$, is the primary source of sample-specific information in the influence calculation. While the first term (impact function gradient) is constant for a given test query and the Hessian is also model-wide, the gradient of the loss for an individual training sample directly reflects how that sample contributes to the model's parameter updates during training.

The core technical idea of OGA is based on the hypothesis that detrimental samples, which negatively impact model utility, will exhibit anomalous behavior in their respective loss gradients. Specifically, their gradients will appear "out of distribution" compared to the gradients of beneficial samples. This is a powerful simplification because computing the gradient of the loss function for each training sample ($\nabla_{\theta} L(x_j, \hat{\theta})$) is a standard operation during model training (backpropagation) and is computationally much cheaper than calculating second-order derivatives or inverting large matrices.

The OGA pipeline proceeds as follows:

1. Gradient Extraction: For each training sample $x_j$ in the dataset, compute its loss gradient with respect to the model parameters, $\nabla_{\theta} L(x_j, \hat{\theta})$. This can be done efficiently in a single pass during or after training. These gradients are high-dimensional vectors, each representing the "direction of influence" of that specific sample on the model's parameters.

1. Gradient Space Construction: These extracted gradient vectors are then treated as data points in a high-dimensional feature space. The dimensionality of this space is equal to the number of model parameters.

1. Outlier Detection: A suitable outlier analysis algorithm is applied to this gradient space. While the talk doesn't specify a particular algorithm, common choices in outlier detection include:

• Isolation Forest: An ensemble method that "isolates" anomalies by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.
• Local Outlier Factor (LOF): Measures the local density deviation of a data point with respect to its neighbors. Anomalies are expected to have a substantially lower density than their neighbors.
• DBSCAN (Density-Based Spatial Clustering of Applications with Noise): Identifies clusters of varying shapes and densities in data, marking points that lie alone in low-density regions as outliers.
• One-Class SVM: A kernel-based method that learns a decision boundary around the "normal" data points, marking anything outside as an anomaly.

The choice of algorithm might depend on the specific characteristics of the gradient distribution and computational constraints, but the underlying principle remains the same: identify points that are statistically unusual or deviate significantly from the majority of gradient vectors.

1. Sample Pruning: Once the outlier gradients are identified, the corresponding training samples $x_j$ are flagged as detrimental. These samples are then removed from the training dataset.

1. Model Retraining: Finally, the deep learning model is retrained on the reduced, "cleaned" dataset. The expectation is that by removing the detrimental samples, the model will learn more effectively, leading to improved performance metrics such as higher accuracy, lower loss, or reduced undesirable behaviors.

In contrast to TracIn, which merely computes the inner product (a measure of cosine similarity if normalized) between gradients, OGA goes a step further by performing a statistical analysis of the gradient distribution. TracIn's assumption of an identity Hessian implies that all parameter dimensions are equally important and uncorrelated, which is rarely true for complex deep learning models. OGA, by identifying genuine statistical outliers, implicitly captures more nuanced deviations in the gradient space that TracIn might miss, without incurring the cost of explicit Hessian computation. This makes OGA a compelling balance between computational efficiency and analytical power for data valuation.

Experimental Setup & Results

▶ Watch: Gradient-Only Approaches and TracIn Limitations (5:20)

The effectiveness of Outlier Gradient Analysis (OGA) was rigorously evaluated across a range of datasets and model architectures, demonstrating its versatility and superior performance compared to existing baselines.

1. Synthetic Data Verification:

The initial validation of OGA's core hypothesis – that detrimental samples exhibit out-of-distribution gradients – was performed on synthetic datasets.

• Datasets:
• A linearly separable dataset, where a subset of samples was intentionally mislabeled (label flipping) to simulate noise, marked as 'X'.
• A non-linearly separable half-moons dataset, also corrupted with mislabeled samples.
• Methodology: Models were trained on these datasets, and the gradients of all training samples were extracted and visualized in a reduced-dimensional space.
• Results: Visualizations clearly showed that the gradients corresponding to the noisy, mislabeled samples consistently clustered away from the gradients of the correctly labeled samples, appearing as distinct outliers. This empirical observation strongly supported the hypothesis, providing a robust foundation for the OGA approach. Removing these detected outliers demonstrably improved classification performance on these simple tasks.

2. Deeper Models and Noisy Learning:

To assess OGA's performance in more realistic and complex scenarios, experiments were conducted on deeper convolutional neural networks and datasets known for containing real-world noise.

• Models: ResNet-34 and ResNet-18, widely used architectures for image classification.
• Datasets: Noisy versions of CIFAR datasets. Crucially, this noise was not synthetic but originated from human annotators (e.g., Amazon MTurkers), reflecting real-world labeling inaccuracies.
• Baselines: OGA was compared against various influence estimation baselines (e.g., TracIn) and other methods specifically designed for noisy learning.
• Metrics: Standard classification performance metrics such as accuracy and loss reduction.
• Results: OGA consistently achieved "top performance" when compared to other influence estimation methods. Furthermore, it proved "very competitive" even against specialized noisy learning baselines. Examples of predicted noisy samples revealed clear mislabeling errors made by human annotators, confirming OGA's ability to identify genuine data quality issues.

3. Synthetic Noise in Natural Language Processing (NLP):

The applicability of OGA extends beyond computer vision to NLP tasks, as demonstrated by experiments with transformer models.

• Model: RoBERTa, a robustly optimized BERT pretraining approach.
• Datasets: GLUE datasets, a collection of natural language understanding tasks.
• Scenario: 20% of the labels in the GLUE datasets were synthetically flipped to introduce controlled noise.
• Results: Even in this NLP setting with a powerful transformer model, OGA achieved "superlative performance" compared to other baselines, showcasing its generalizability across different data modalities and model types.

4. Large Language Models (LLMs) and Influential Sample Identification:

A particularly compelling application of OGA was demonstrated in the context of LLMs, where identifying influential samples is crucial for debugging and understanding model behavior.

• Task: Fine-tuning an LLM and then, for a given test prompt, identifying the single most influential training or fine-tuning sample that contributed to the model's response.
• Benchmarks: The benchmarks used were "fairly simple," characterized by test prompts and their most influential samples having "very high similarities" or being "nearly identical." This simplicity was intentionally chosen to establish a clear ground truth for evaluation.
• Results: Despite the conceptual complexity of LLMs, OGA achieved "near-perfect performance" in identifying these influential samples. This suggests significant potential for OGA in LLM debugging and data curation, although the authors acknowledge the need for more sophisticated benchmarks to fully capture real-world LLM complexities.

Across all experimental settings, OGA consistently demonstrated its capability to efficiently and accurately identify detrimental samples, leading to tangible improvements in model performance by effectively pruning noisy or problematic data from training sets.

Practical Implications

▶ Watch: Key Observations for Detrimental Sample Identification (6:45)

The introduction of Outlier Gradient Analysis (OGA) carries significant practical implications for various stakeholders in the machine learning ecosystem, from data scientists and model builders to infrastructure teams and deployers.

For Practitioners and Model Builders:

• Enhanced Model Performance: The most direct benefit is the ability to train models with higher accuracy and reduced loss. By efficiently identifying and removing detrimental samples, OGA enables cleaner datasets, leading to models that generalize better and are more robust. This is particularly valuable in domains where data quality is inherently challenging, such as medical imaging, social media analysis, or user-generated content.
• Effective Model Debugging: OGA offers a powerful tool for understanding and rectifying unwanted model behaviors. As illustrated by the GPT-3.5 Turbo Slack bot example (where the model failed to execute a command despite repeated prompts), LLMs can learn incorrect patterns from fine-tuning data. Instead of resorting to expensive and time-consuming alignment training methods like RLHF (Reinforcement Learning from Human Feedback) or SFT (Supervised Fine-Tuning), OGA can pinpoint the specific training samples that contributed to these undesirable traits. This allows for targeted data removal or correction, making the debugging process more efficient and precise.
• Resource Efficiency: By enabling the pruning of training datasets, OGA can lead to more lightweight models. A smaller, higher-quality dataset means faster training times and potentially smaller model sizes if the pruning is substantial. This translates to reduced computational costs (GPU/TPU hours) and lower energy consumption, which are increasingly important considerations for sustainable AI.

For Infrastructure Teams and Data Providers:

• Framework for Data Compensation: In the near future, OGA could form the backbone of systems designed to fairly compensate data providers. By efficiently valuing individual data samples – giving higher weight to those that positively influence model performance and penalizing those that are detrimental – platforms can incentivize the submission of high-quality data. This fosters a more equitable and efficient data economy. The example of distinguishing between a correct answer ("White House in Washington DC") and an incorrect one ("White House in Japan") for a query like "Where does the US president live?" highlights this potential.
• Streamlined Data Curation: Data preparation and cleaning are often the most time-consuming parts of an ML project. OGA automates a crucial aspect of this by programmatically identifying problematic samples, reducing the manual effort required for data inspection and annotation.

Tradeoffs and Limitations:

While OGA offers substantial advantages, it's important to acknowledge its limitations and the tradeoffs involved:

• Assumptions and Context-Dependency: OGA's efficacy relies on the fundamental assumption that detrimental samples are relatively few and exhibit gradients that are genuinely "out-of-distribution." This assumption holds well in traditional noisy learning scenarios (e.g., mislabeled images). However, as highlighted in the Q&A, in complex scenarios like the Slack bot where model behavior is "incorrect" but the data isn't necessarily "noisy" or "mislabeled" in the conventional sense, OGA's applicability might be limited. The "real answers" in such cases might themselves be rare or OOD, challenging the core premise.
• Defining "Detrimental": For nuanced LLM behaviors, defining what constitutes a "detrimental" sample can be complex. OGA currently excels at identifying samples leading to clear performance degradation (e.g., misclassification). For subtle behavioral issues, more sophisticated benchmarks and perhaps a more context-aware definition of "outlier" in the gradient space might be necessary.
• Impact on Rare but Important Data: A critical question raised during the talk concerned whether OGA might inadvertently filter out rare data points that are crucial for generalization, especially those on the "long tail" of the data distribution. The speaker acknowledged this hasn't been directly tested but suggested that semi-supervised outlier algorithms combined with similarity matching (similar to TracIn) could potentially address this by identifying relevant rare samples rather than simply removing all outliers. This represents an area for future research.
• Benchmarking Complexity: While OGA performed well on "fairly simple" LLM benchmarks, real-world LLM debugging scenarios are far more complex. Developing more robust and realistic benchmarks that capture the full spectrum of undesirable LLM behaviors will be essential to fully validate and advance OGA's utility in this domain.

In summary, OGA provides a highly efficient and effective mechanism for improving data quality and model performance. Its gradient-centric approach sidesteps historical computational bottlenecks, making advanced data valuation techniques accessible for a broader range of deep learning applications. However, continued research is needed to refine its application in highly complex, context-dependent scenarios, particularly within the rapidly evolving landscape of large language models.

Key Takeaways

• Data Valuation is Critical: Understanding the impact of individual training samples is crucial for improving model performance, debugging unwanted behaviors, and fairly compensating data providers.
• Hessian Bottleneck Overcome: Traditional influence functions, while theoretically sound, are computationally intractable due to the inverse Hessian matrix. OGA circumvents this by focusing on cheaper-to-obtain first-order gradients.
• Outlier Gradient Hypothesis: Detrimental training samples are hypothesized to be small in number and exhibit gradients that are statistically out-of-distribution compared to beneficial samples. This hypothesis was empirically verified on synthetic data.
• Efficient Identification with OGA: Outlier Gradient Analysis (OGA) leverages standard outlier detection algorithms in the gradient space to efficiently identify these detrimental samples without costly second-order computations.
• Proven Performance Across Modalities: OGA demonstrated superior or competitive performance in identifying noisy samples across diverse tasks and models, including ResNet-18/34 on noisy CIFAR, RoBERTa on noisy GLUE, and LLMs for influential sample identification.
• Practical Benefits & Future Scope: OGA offers significant practical benefits for model debugging, performance enhancement, and potentially fair data compensation. However, its application to complex, nuanced LLM behaviors and the challenge of preserving rare but important data require further research and more sophisticated benchmarks.

About the Speaker(s)

The primary speaker for this presentation was Anshuman Chhabra, who is an Assistant Professor at the University of South Florida. He presented this work as a joint effort with his collaborators: Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu. Unfortunately, his co-authors were unable to attend the conference.