The Hidden Bloat in Machine Learning Systems

The Hidden Bloat in Machine Learning Systems

Speakers: Huaifeng Zhang, PhD Student, Chalmers University; Ahmed Ali-Eldin, Supervisor, Chalmers University

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The proliferation of machine learning (ML) frameworks like PyTorch and TensorFlow has driven rapid innovation, but this growth comes with an often-overlooked cost: software bloat. This talk, presented by Huaifeng Zhang and his supervisor Ahmed Ali-Eldin from Chalmers University, addresses the critical issue of unnecessary code within these frameworks, particularly focusing on their shared libraries. Drawing an analogy to Michelangelo's David, where the artist "removed everything that is not David," the presenters introduce their novel approach, Negativia Amal, which aims to de-bloat ML frameworks by intelligently removing unneeded code from shared libraries.

The core problem stems from the continuous addition of features and functionalities by developers, leading to massive codebases. For instance, PyTorch alone sees hundreds of thousands of new lines of code annually. While this enriches the frameworks, it also inflates their size, creating significant challenges for deployment in resource-constrained environments like edge devices and mobile platforms. Negativia Amal specifically targets "type two bloat," a category of unnecessary features or large libraries that can be customized and removed based on specific use cases and workloads, offering a tailored solution to this pervasive problem.

This work is particularly pertinent as ML models become increasingly ubiquitous, demanding efficient and lean deployments. The substantial size of ML shared libraries, often hundreds of megabytes larger than those in traditional applications, combined with their unique blend of CPU and GPU code, presents a complex de-bloating challenge. Negativia Amal's contribution lies in being the first tool to specifically tackle GPU code bloat, which constitutes a significant portion of these libraries, promising substantial reductions in file size, memory footprint, and improved loading performance.

Background

▶ Watch: Introduction to de-bloating machine learning frameworks (0:00)

The rapid evolution and widespread adoption of machine learning have led to the growth of sophisticated frameworks such as PyTorch and TensorFlow. These frameworks, written in a combination of languages like C++, CUDA, and Python, are designed to be general-purpose, supporting a vast array of models, architectures, and deployment scenarios. While this versatility is a strength, it inadvertently fosters a significant problem: software bloat. This phenomenon refers to the inclusion of unnecessary code, features, or large libraries that are not utilized in a specific application but still contribute to its overall size and resource consumption.

Prior research has categorized software bloat into different types. "Type one bloat" refers to universally removable dead code, while "type two bloat" pertains to code that is unnecessary for customized use cases or specific workloads. The latter is the focus of this paper, recognizing that most users only leverage a fraction of a framework's extensive capabilities. The implications of this bloat are particularly severe in the context of ML systems. Unlike traditional applications, which typically have shared libraries ranging from 2 MB to 30 MB, ML applications often incorporate shared libraries that balloon to sizes between 200 MB and 1200 MB. This colossal difference, highlighted by the authors' prior work, underscores the unique challenges faced by ML deployments.

A critical distinguishing factor for ML shared libraries is their dual nature, encompassing both CPU code and GPU code. In contrast, most generic applications primarily consist of CPU code. The authors' investigation revealed that GPU code constitutes a substantial portion of ML shared libraries, ranging from 68% to 91% of their total size. This indicates that while CPU code bloat has received some attention (e.g., through tools like Negativia for CPU de-bloating, mentioned in the talk as a precursor), the bloat within the GPU code domain remains largely unexplored and unaddressed.

The challenges in tackling GPU code bloat are multifaceted. Firstly, many critical machine learning shared libraries are not open source, denying direct access to their underlying source code for analysis and modification. Secondly, the area of GPU code de-bloating is inherently underexplored, lacking established methodologies and tools. Finally, the internal structure and compilation mechanisms of GPU code, particularly CUDA code, differ significantly from conventional CPU code, demanding specialized approaches for identification and removal of unused components. These factors collectively establish a clear need for a novel solution capable of intelligently pruning the GPU code within ML shared libraries, a gap that Negativia Amal aims to fill.

Key Findings

▶ Watch: Problem of huge shared library sizes in ML (4:00)

The research presented on Negativia Amal unveils several significant findings that collectively highlight the scale of software bloat in machine learning frameworks and the effectiveness of their proposed de-bloating methodology.

Firstly, Negativia Amal is established as the first dedicated tool to specifically address GPU code bloat within machine learning shared libraries. This is a crucial distinction, given the prior lack of solutions targeting the dominant GPU code component of these frameworks, which can account for up to 91% of library size. Its novel approach extends beyond CPU-only de-bloating, providing a holistic solution for modern ML systems.

The most striking finding is the substantial reduction in various metrics achieved by Negativia Amal. Across diverse workloads and frameworks, the tool demonstrated an average 66% reduction in overall file size of the shared libraries. This translates directly into more compact deployments. Furthermore, it led to significant memory savings: approximately 75% reduction in CPU memory usage and 70% reduction in GPU memory usage. These memory optimizations are critical for deploying models on edge devices, mobile platforms, or in cloud environments where resource efficiency is paramount. Beyond static size and memory, Negativia Amal also delivered a remarkable 44% improvement in execution time, specifically related to library loading performance, which can dramatically reduce cold start latencies in serverless or on-demand inference scenarios. On average, across all evaluated workloads, this amounted to a reduction of around 2.5 GB in CPU memory, 443 MB in GPU memory, and 2.6 seconds in execution time.

A key insight into the root causes of bloat emerged from the analysis of GPU code reduction. The authors discovered that a staggering 80% of GPU code bloat stemmed from a mismatch in GPU architectures. This means that a large proportion of cubins (CUDA binary units containing compiled GPU code) are included in the shared library but are incompatible with the target GPU architecture. The two primary reasons identified were: the cubin simply not matching the GPU architecture, or the cubin matching the architecture but containing no used kernels for the specific workload. This finding underscores that software bloat is not solely a software development issue but can also originate from hardware-related compilation and distribution strategies.

Finally, the evaluation also revealed a notable degree of commonality in function usage across different workloads. For instance, when analyzing the torch CUDA shared library with six distinct workloads spanning PyTorch, VLM, and Transformers architectures, it was found that 77% of the used functions were identical. This indicates that while specific model architectures or tasks might vary, a significant core set of GPU functions is commonly invoked. This observation suggests potential for more generalized de-bloating strategies or optimized base libraries, even as the tool provides workload-specific customization.

Technical Deep Dive

▶ Watch: Identifying the challenge: unexplored GPU code bloat (7:00)

Negativia Amal is a sophisticated de-bloating tool designed to prune both CPU and, crucially, GPU code within machine learning shared libraries. Its workflow is orchestrated in two primary phases: Kernel Detection and Kernel Location, followed by the actual removal process. The tool's innovation lies in its ability to identify and remove unused GPU code without requiring access to the original source code, making it applicable to proprietary libraries.

The process begins with Kernel Detection. The fundamental insight here is that GPU kernels are ultimately launched by the CPU. A CPU launching kernel is a kernel directly invoked by CPU code. These CPU-launched kernels, in turn, can launch other kernels on the GPU, which are referred to as GPU launching kernels. Together, these form a kernel call graph, where the CPU launching kernel acts as the entry point. To detect these used kernels, Negativia Amal employs an elegant monitoring technique. It hooks into the CUDA API function CU module get function. This function is invariably called by the CPU code when it intends to invoke a GPU kernel. By intercepting the input to this function, Negativia Amal can reliably extract the name of the kernel being launched from the CPU. These identified kernels are then marked as "used." It's important to note a limitation at this stage: while CPU launching kernels are detected, the tool cannot directly detect GPU launching kernels solely through this CPU-level monitoring.

Following kernel detection, the workflow proceeds to Kernel Location. This phase aims to pinpoint the specific file ranges within the shared library that correspond to the identified used kernels, enabling their retention while removing the rest. To achieve this, Negativia Amal first analyzes the internal structure of GPU code as specified by the CUDA documentation. GPU code is organized hierarchically: it consists of a set of regions, each containing a set of elements. Crucially, each element contains a cubin (CUDA Binary) – the actual compiled GPU code – along with an element header.

The core logic for removal operates at the cubin level. For each cubin present in the shared library, Negativia Amal extracts the kernels included within it. The decision to retain or remove a cubin is based on two primary criteria:

1. Used CPU Launching Kernel Inclusion: If a cubin contains at least one of the CPU launching kernels identified in the detection phase, that entire cubin is marked for retention.
2. Handling GPU Launching Kernels: A critical design choice addresses the challenge of implicitly detecting GPU launching kernels. The authors leverage the observation that if one kernel launches another kernel (i.e., a GPU launching kernel scenario), these two kernels are typically compiled into the same cubin. Therefore, by retaining the entire cubin that contains the initial CPU launching kernel, the entire associated kernel call graph, including any subsequent GPU launching kernels, is automatically preserved. This avoids the need for complex, runtime GPU-level call graph analysis, which would be significantly more challenging, especially for closed-source libraries.
3. GPU Architecture Matching: An additional layer of filtering is applied. A cubin is only retained if it not only contains a used CPU launching kernel but also matches the target GPU architecture. Cubins that do not match the target architecture, even if they contain some kernels, are deemed irrelevant for that specific deployment and are removed. This mechanism is responsible for a large portion of the observed bloat reduction, as frameworks often ship with compiled code for a wide range of GPU architectures.

The de-bloating process essentially identifies and retains only the cubins that are essential for the specific workload and target architecture, discarding all other cubins. It's crucial to understand that Negativia Amal operates by removing entire cubins; it does not perform granular removal of individual kernels within a cubin, nor does it repackage or recompile cubins. The output is a specialized shared library that contains only the necessary compiled code for the given workload. This runtime-based approach means users execute their workload, Negativia Amal observes the kernel usage, and then generates a de-bloated library tailored to that specific execution profile.

Experimental Setup & Results

▶ Watch: Overview of Negativia Amal's GPU de-bloating workflow (9:00)

To thoroughly evaluate the efficacy of Negativia Amal, the authors conducted extensive experiments across a diverse set of popular machine learning frameworks and operational modes. The experimental setup was designed to quantify the reduction in file size, memory footprint, and execution time, as well as to identify the primary sources of bloat.

Frameworks and Workloads: The evaluation utilized four prominent machine learning frameworks: PyTorch, TensorFlow, VLM (Vision-Language Models), and Transformers. Within these frameworks, various models and modes of operation were tested, including both training and inference workloads. This comprehensive approach ensured that the findings were generalizable across different ML applications and use cases. The scale of the evaluation was significant, involving the analysis and de-bloating of over 300 shared libraries, demonstrating the robustness of Negativia Amal.

Bloat Identification and Reduction:

The initial phase of evaluation focused on identifying the magnitude and sources of bloat. Negativia Amal achieved an average 66% reduction in the overall file size of the shared libraries. Breaking this down, the analysis revealed that the majority of the original library size was contributed by GPU code. A pivotal finding was that approximately 80% of the GPU code was removed. This substantial reduction was primarily attributed to two reasons:

1. GPU Architecture Mismatch: A significant portion of the cubins (compiled GPU code units) included in the libraries did not match the specific GPU architecture being used for the workload. This highlights a common practice in framework distribution, where a single library bundles code for multiple architectures to ensure broad compatibility, leading to considerable bloat for any single target device.
2. Unused Kernels within Matching Cubins: Even for cubins that matched the GPU architecture, many were found not to contain any kernels that were actually invoked by the specific workload. This points to the "type two bloat" – features or functionalities that are present but simply not utilized by the user's application.

In essence, the experiments concluded that the largest proportion of cubins were deleted due to an architecture mismatch, underscoring that software bloat can significantly stem from hardware-agnostic compilation and distribution strategies.

Influence of Different Workloads:

To understand how different tasks affect the de-bloating potential, the authors investigated the overlap of used functions across various workloads. For instance, the torch CUDA shared library was evaluated with six distinct workloads, encompassing models from PyTorch, VLM, and Transformers architectures. The key observation was that 77% of the functions used by these diverse workloads were common functions. This suggests that while de-bloating is workload-specific, there's a substantial core set of functions that are universally invoked, potentially allowing for some level of generalized optimization or a base de-bloated library.

Runtime and Memory Performance:

The most impactful results were observed in the improvements to runtime performance and memory consumption:

• Memory Reduction: Negativia Amal achieved an impressive 75% reduction in CPU memory usage and a 70% reduction in GPU memory usage. These reductions are critical for enabling larger models or more concurrent inference requests within fixed memory budgets, especially on resource-constrained devices.
• Execution Time Improvement: The de-bloating process led to a 44% reduction in execution time. This improvement primarily reflects faster library loading times, which can significantly enhance the responsiveness of ML applications, particularly in serverless or cloud-based inference scenarios where cold start latencies are a major concern.

Quantitatively, the average reductions across all evaluated workloads were substantial: approximately 2.5 GB of CPU memory, 443 MB of GPU memory, and 2.6 seconds in execution time. These headline numbers unequivocally demonstrate the practical benefits of Negativia Amal in making machine learning deployments more efficient and performant.

Practical Implications

▶ Watch: Kernel Locator: Identifying and removing unused GPU code (12:00)

The findings and capabilities of Negativia Amal carry significant practical implications for a wide spectrum of stakeholders in the machine learning ecosystem, from individual practitioners to large-scale infrastructure providers.

For model builders and deployers, Negativia Amal offers a crucial tool for optimizing the footprint of their ML applications. In environments with strict memory constraints, such as edge devices, mobile phones, or embedded systems, the ability to reduce shared library sizes by 66% and GPU memory usage by 70% can be the difference between a feasible deployment and an impossible one. This allows for the deployment of more complex models or a greater number of models concurrently within a given hardware budget. Furthermore, for serverless inference functions or other on-demand ML services, the 44% reduction in execution time, primarily due to faster library loading, translates directly into reduced cold start latencies. This improves user experience and can lead to cost savings in cloud environments where billing is often tied to compute duration.

Infrastructure teams and cloud providers stand to benefit immensely. Cloud platforms that allow users to upload custom serverless functions often face challenges with large, generic ML framework dependencies. Negativia Amal provides a mechanism for these providers to automatically specialize and optimize user-uploaded functions. Instead of requiring users to recompile frameworks from source (a complex and often infeasible task), providers can use Negativia Amal to de-bloat libraries based on the specific function's workload, reducing resource consumption and improving overall platform efficiency. This not only lowers operational costs for the provider but also enhances the performance and scalability of their serverless offerings.

However, the approach also presents certain tradeoffs and limitations. The de-bloated library is highly specialized for the specific workload it was observed during. If the application's functionality or the models it uses change significantly, a new de-bloating pass might be required. This introduces a management overhead for maintaining specialized library versions. From a framework vendor's perspective, the challenge lies in balancing generality with specialization. Vendors aim to support the broadest possible set of use cases, making it difficult to pre-de-bloat their distributions without alienating large segments of their user base. The talk highlights this "gap between library developers and library users," where developers build for generality, but users only need a specific subset.

Another important consideration, brought up in the Q&A, is the impact on very large models. For models with hundreds of billions of parameters (e.g., DeepSeek 600GB), the model's parameters (data) dominate the memory footprint, not the code. In such scenarios, while code de-bloating still provides benefits in terms of file size and loading performance, its relative impact on total memory consumption might be less pronounced compared to smaller models where the code constitutes a larger proportion of the overall memory usage. Finally, the technical limitation of de-bloating at the cubin level rather than individual kernels means that if a cubin contains one used kernel and many unused ones, the entire cubin is still retained. While this simplifies the process and addresses the challenge of non-open-source libraries, it implies there might be further, albeit more complex, opportunities for optimization at a finer granularity.

Despite these considerations, Negativia Amal represents a significant step forward in making machine learning systems more efficient, particularly in an era where ML is increasingly deployed in diverse and resource-constrained environments.

Key Takeaways

• Significant Bloat in ML Shared Libraries: Machine learning frameworks like PyTorch and TensorFlow carry substantial unnecessary code, particularly in their shared libraries, which can be 10-50 times larger than those in traditional applications.
• GPU Code Dominates Bloat: GPU code accounts for 68-91% of the size of ML shared libraries, making it a critical, yet previously underexplored, target for de-bloating.
• Negativia Amal: First GPU De-bloating Tool: The presented Negativia Amal is the first tool designed to specifically address GPU code bloat in ML libraries, operating without requiring access to source code.
• Impressive Resource Reductions: The tool achieves an average 66% file size reduction, 75% CPU memory reduction, 70% GPU memory reduction, and 44% execution time improvement (primarily loading performance).
• Architecture Mismatch is Key Bloat Source: A major cause of GPU code bloat (80% of removed GPU code) is the inclusion of cubins that do not match the target GPU architecture, highlighting a hardware-related aspect of software bloat.
• Target Users: End-Users and Cloud Providers: Negativia Amal is most beneficial for end-users deploying specialized ML models and cloud providers optimizing serverless ML functions, enabling more efficient deployments on resource-constrained or on-demand platforms.

About the Speaker(s)

The talk was presented by Huaifeng Zhang, a PhD student in the Computer Science and Engineering (CSE) department at Chalmers University. Huaifeng is actively engaged in cutting-edge research in machine learning systems optimization and is currently on the job market seeking internship opportunities. His work on Negativia Amal demonstrates a deep understanding of the complexities of ML framework internals and a practical approach to addressing system-level challenges.

He was accompanied by his supervisor, Ahmed Ali-Eldin, also from Chalmers University. Ahmed's involvement underscores the academic rigor and foundational research behind Negativia Amal, building upon prior work in software de-bloating. Both speakers were available for questions, even at 3 AM in Sweden, highlighting their dedication to the research and engagement with the ML community.