A Bring-Your-Own-Model Approach for ML-Driven Storage Placement in Warehouse-Scale Computers

A Bring-Your-Own-Model Approach for ML-Driven Storage Placement in Warehouse-Scale Computers

Speakers: Chenxi Yang, Yan Li, Martin Maas, Mustafa Uysal, Arif Merchant, Richard McDougall

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

In the sprawling landscape of modern data centers, where "warehouse-scale computers" are the norm, storage systems represent a significant portion of the total operational cost. Within these complex infrastructures, the decision of where to place data – specifically, whether to store it on fast, expensive Solid State Drives (SSDs) or slower, cheaper Hard Disk Drives (HDDs) – is a critical factor profoundly impacting both performance and cost. This talk, presented by Chenxi Yang and co-authored with Yan Li, Martin Maas, Mustafa Uysal, Arif Merchant, and Richard McDougall, introduces a novel Bring Your Own Model (BYOM) approach to machine learning (ML)-driven storage placement, aiming to optimize this crucial decision.

The research directly addresses the practical challenges that typically hinder the deployment of monolithic ML models in production storage systems, particularly the rapid and dynamic shifts in real-world workloads and the high overhead associated with passing extensive application-layer information to the storage layer. By proposing a decomposed, multi-layer design, the BYOM approach empowers individual applications with their own lightweight ML models. These models generate concise "hints" that guide an adaptive algorithm at the storage layer, leading to more agile, efficient, and cost-effective data placement.

This work is significant because it not only demonstrates substantial cost savings—over 2x in simulation and more than 4x in experimental production—but also achieves this with remarkably low inference latency, making ML-driven storage optimization truly practical at Google's immense scale. The presented methodology offers a blueprint for overcoming common deployment hurdles in ML for systems, suggesting a broader applicability beyond just storage to other critical areas like resource management and network control.

Background

▶ Watch: Introduction to ML-driven storage placement problem (0:00)

The evolution of modern data centers into warehouse-scale computing environments has made storage systems a primary driver of infrastructure costs. Effective data management, particularly the data placement problem, is paramount. This involves judiciously assigning data to different tiers of storage hardware, primarily fast but expensive SSDs, and slower but cheaper HDDs. The goal is to maximize performance while minimizing cost, a classic optimization challenge.

Historically, two main categories of methods have been employed for data placement:

1. Heuristics: These involve manually designing rules based on observed historical usage patterns. While heuristics are generally fast and interpretable, their performance often proves suboptimal as they struggle to capture complex, non-linear relationships in data access patterns.
2. Machine Learning (ML) based approaches: These leverage big data to identify hidden patterns and make more informed placement decisions. ML methods hold the promise of superior decision-making given a clear objective. However, they come with their own set of practical challenges: defining a precise optimization objective can be complex, ML models introduce runtime latency, and their decisions often lack the interpretability inherent in rule-based systems.

Existing ML-based approaches often assume a monolithic model that attempts to solve the data placement problem end-to-end. Such models typically extract a vast array of application features from the application layer and then directly infer placement decisions at the storage layer. While conceptually appealing, deploying these monolithic models in production environments faces significant practical hurdles:

• Dynamic Workloads: Real-world workloads exhibit extreme dynamism, with usage patterns shifting rapidly. Analysis of Google's production traces reveals substantial variations in resource usage (e.g., space, lifetime) both within individual workloads over time and across different workloads. This volatility means that a large, static monolithic model struggles to adapt quickly enough to maintain optimal performance and cost efficiency, as workload update cycles are often faster than the system's ability to retrain and deploy new models.
• Expensive Information Overhead: For a monolithic model to make informed decisions, it often requires a rich set of features from the application layer. Passing all this detailed information down to the storage layer introduces considerable overhead in terms of network bandwidth, processing, and latency, making real-time deployment economically and technically challenging.

This work specifically aims to overcome these limitations of ML-based approaches, making them more adaptable, efficient, and practical for deployment in highly dynamic, warehouse-scale storage systems.

Key Findings

▶ Watch: Introducing the "Bring Your Own Model" (BYOM) key idea (5:00)

The core contribution of this research is the introduction of the Bring Your Own Model (BYOM) concept, a paradigm shift for ML-driven system optimization. Instead of relying on a single, monolithic ML model, the BYOM approach proposes a decomposition strategy:

• Decentralized ML for Application Specificity: The system breaks down the complex, end-to-end storage placement problem into smaller, more manageable sub-problems. This allows for the deployment of application-specific models, where each individual workload or application can have its own tailored ML model. These models are designed to capture the unique characteristics and access patterns of their respective applications, providing more precise and adaptable predictions.
• Cross-Layer Design with Adaptive Algorithm: The application-specific models operate at the application layer, predicting lightweight "hints" (e.g., predicted IO density rankings). These hints are then passed efficiently down through a minimal protocol to the storage layer. Crucially, at the storage layer, a co-designed custom adaptive algorithm interprets these hints in conjunction with real-time hardware utilization data to make dynamic placement decisions. This multi-layer approach embraces the inherent architectural hierarchy of modern storage systems.
• Significant Cost Savings: The BYOM methodology demonstrated remarkable economic benefits. In large-scale simulations using real Google production traces across all Google Cloud clusters, the proposed method achieved over 2x average Total Cost of Ownership (TCO) savings compared to state-of-the-art baselines. Furthermore, in an end-to-end experimental production integration within Google's data processing framework, the system delivered more than 4x total cost savings. The speaker emphasized that even a 1% TCO saving at Google's scale translates into substantial financial impact.
• Low Operational Latency: A critical practical finding was the low overhead introduced by the ML component. The average latency per inference for the application-specific models was measured at approximately 4 milliseconds in an online production setup, which is considered "pretty reasonable" for real-time system operations. This low latency is largely attributed to the use of simpler, efficient models for per-application prediction.
• Generalizable Design Principle: Beyond its immediate application to storage, the talk highlights that this cross-layer design principle, which combines application-specific ML with adaptive system algorithms, is a powerful and generalizable method. The researchers believe it can be extended to optimize other complex system applications, including resource management and network controlling, indicating a broader impact on future system design.

Technical Deep Dive

▶ Watch: BYOM design details: Oracle policy and adaptive algorithm (6:30)

The technical ingenuity of the BYOM approach lies in its strategic decomposition of the ML-driven storage placement problem and the co-design of its components across system layers.

At its core, the BYOM architecture replaces a single, monolithic ML model with a distributed intelligence framework. Instead of a large model attempting to understand and predict for all workloads simultaneously, the design introduces application-specific models. Each distinct workload or application is paired with its own dedicated ML model, residing at the application layer. These models are trained to understand the unique characteristics of their specific workload—such as I/O patterns, access frequency, and data lifetimes—and to predict compact, high-level "hints" rather than direct placement decisions.

The crucial aspect of communication between layers is the hints protocol. Instead of transferring vast amounts of raw application-layer telemetry, only these small, pre-processed hints are passed down from the application layer to the storage layer. This minimizes the data transfer overhead, addressing a key challenge of monolithic ML systems. The hints are designed to be highly informative yet concise, effectively summarizing the application's storage requirements. For this specific problem, the models predict ranking buckets for IO density at runtime. This means they classify data into categories based on its predicted I/O intensity, indicating which data would benefit most from faster storage.

Complementing the application-layer models is a co-designed custom adaptive algorithm residing at the storage layer. This algorithm is the "brains" of the placement decision, but it operates reactively and intelligently. It takes the incoming hints (IO density ranking buckets) from the applications and combines them with real-time feedback on the online hardware utilization. For instance, if the system detects that SSD capacity is nearing its limit, the adaptive algorithm might become more selective, admitting only data from the top one or two ranking buckets (i.e., the most I/O-intensive data) onto the SSDs. Conversely, if SSDs have ample free space, it might relax its criteria and admit data from a broader range of ranking buckets. This dynamic adjustment ensures efficient utilization of expensive SSD resources and prevents over-provisioning or under-utilization.

The design of the online models was informed by an optimal policy analysis. The researchers first established a "clairvoyant" optimal placement using a linear programming solver applied to historical workload data. This allowed them to understand the absolute best possible placement decisions given full knowledge of a file's entire lifecycle and resource usage. Their analysis of this optimal policy revealed a strong correlation between IO operation cost relative to a job's storage footprint and lifetimes and the optimal placement. This insight was critical, guiding the online models to predict precisely these relevant features—the ranking buckets for IO density—as their output hints.

Regarding the specific ML models, the application-specific models are implemented as 15-class Gradient Boosted Trees (GBTs). This choice is deliberate and contrasts with the larger Neural Networks or Transformers often employed in monolithic ML approaches for systems. The rationale for using GBTs is their relative simplicity, faster inference times, and smaller model footprint. Since each model only needs to capture the characteristics of a single workload, a complex, general-purpose model is not necessary. The efficiency of GBTs contributes directly to the reported low inference latency of 4 milliseconds, making the BYOM approach viable for real-time production environments.

Experimental Setup & Results

▶ Watch: Large-scale simulation results: significant TCO savings (8:10)

The evaluation of the BYOM approach focused on two crucial aspects: quantifying the cost savings in realistic scenarios and assessing the practical overhead (specifically latency) incurred by integrating machine learning into production systems.

The first phase involved a large-scale simulation setup. This simulation was powered by real production traces from Google, encompassing data from "all the clusters of machines in Google Cloud." This high-fidelity environment ensured that the evaluation accurately reflected the complexities and scale of Google's actual infrastructure. The primary metric for this evaluation was Total Cost of Ownership (TCO) savings. The results demonstrated that the BYOM method achieved over 2x of the cost savings on average compared to existing state-of-the-art approaches. The speaker underscored the significance of these savings, noting that "even 1% of the TCO savings is already very large and can be mapped to tons of dollars given the scale of Google," highlighting the profound economic impact of a 2x improvement.

The second, more rigorous phase involved an end-to-end system integration into an experimental production setup within Google's data processing framework. This deployment provided a real-world testbed for the BYOM approach, moving beyond simulation to a live, operational environment. In this integrated setting, the method showcased even more impressive results, delivering more than 4x of the total cost savings. This substantial improvement validates the practical efficacy and robustness of the BYOM design when faced with actual production workloads and system constraints.

Beyond cost, the practical viability of any ML-driven system hinges on its performance overhead. The experimental integration also meticulously measured the latency introduced by the ML inference process. The results indicated an average latency of approximately 4 milliseconds per inference. This figure was deemed "pretty reasonable in an online production setup," demonstrating that the BYOM approach can deliver significant cost benefits without compromising real-time system responsiveness.

The speaker clarified the definition of TCO used in Google, which is a widely recognized metric typically mapped to dollar costs. This comprehensive cost metric incorporates various factors, including network cost, SSD wear-out, and storage cost, providing a holistic view of the economic benefits achieved. The consistent and substantial cost savings, coupled with low inference latency, underscore the BYOM approach as a highly effective and practical solution for ML-driven storage optimization in warehouse-scale computing.

Practical Implications

▶ Watch: End-to-end system integration: 4X cost savings, low latency (9:20)

The "Bring Your Own Model" approach carries profound practical implications for various roles within warehouse-scale computing environments, offering solutions to long-standing challenges in ML-driven systems.

For Practitioners and Infrastructure Teams:

• Enhanced Adaptability to Dynamic Workloads: The most significant benefit is the ability to cope with highly dynamic and rapidly shifting workloads. Instead of a slow, monolithic retraining cycle, individual application-specific models can be updated or retrained much faster and more frequently. This localized agility allows infrastructure teams to maintain optimal performance and cost efficiency for each workload without disrupting the entire system, a critical capability in environments like Google Cloud.
• Reduced Operational Overhead: By generating and passing only compact "hints" from the application layer to the storage layer, the BYOM approach drastically cuts down on the network and processing overhead typically associated with moving large feature sets across system boundaries. This makes ML integration less resource-intensive and more scalable.
• Fine-Grained Optimization: Each application's unique access patterns and performance requirements can be precisely captured by its dedicated model, leading to more tailored and effective placement decisions. This contrasts sharply with global models that might struggle to find a single optimum across diverse workloads.

For Model Builders:

• Simpler Model Development: The ability to use simpler, more interpretable models like Gradient Boosted Trees for individual applications simplifies the entire model lifecycle, from development and training to debugging and maintenance. This reduces the complexity and specialized expertise required compared to building and managing large Neural Networks or Transformers that must generalize across all workloads.
• Focused Problem Scope: Model builders can concentrate on optimizing a specific application's model, understanding its nuances without the added burden of accounting for the complexities and interactions of all other workloads in the system.

For Deployers and System Architects:

• Cross-Layer Design Paradigm: The BYOM framework provides a powerful architectural pattern for integrating ML into complex, multi-layered systems. It offers a blueprint for how intelligence can be distributed and communicated effectively across abstraction layers, potentially serving as a model for other system optimization problems beyond storage.
• Proven Cost-Effectiveness and Performance: The demonstrated 2x to 4x cost savings, combined with a low 4ms inference latency, provides a strong business case for adopting ML-driven solutions for storage placement, validating the investment in such a system.

Tradeoffs and Limitations:

• Model Management Complexity: While individual models are simpler, managing a potentially large number of application-specific models (e.g., hundreds or thousands) could introduce new operational complexities related to model versioning, deployment pipelines, monitoring, and lifecycle management. The talk does not delve into these aspects, but they are inherent considerations for large-scale adoption.
• Initial Data and Training Requirements: For every new application or significant workload change, there might be a need for initial data collection and dedicated model training. This upfront investment could be a barrier for rapidly evolving or highly ephemeral workloads if automation for model lifecycle management is not robust.
• Inter-Workload Dependencies: The BYOM approach optimizes placement primarily at the per-application level. While the adaptive algorithm at the storage layer considers global hardware utilization, there could still be subtle inter-workload dependencies or emergent system-wide behaviors that are not fully captured by local optimizations. However, the adaptive layer provides a crucial global协调 mechanism.
• Generalizability of Hints: The effectiveness of BYOM relies on identifying a sufficiently compact and informative "hint" (like IO density ranking buckets) that can effectively guide the lower system layers. Designing such hints for other system optimization problems might not always be straightforward.

Key Takeaways

• Machine learning offers significant potential for optimizing storage placement in warehouse-scale computers, but traditional monolithic ML models struggle with dynamic workloads and high information transfer overhead.
• The Bring Your Own Model (BYOM) approach revolutionizes ML-driven system design by decomposing the problem, allowing lightweight, application-specific models to operate at the application layer.
• These application-specific models predict concise "hints" (e.g., IO density ranking buckets) that are efficiently passed to the storage layer, where a co-designed adaptive algorithm makes real-time placement decisions based on these hints and current hardware utilization.
• This cross-layer design achieved impressive results: over 2x Total Cost of Ownership (TCO) savings in large-scale simulation and over 4x total cost savings in an experimental production integration within Google's data processing framework.
• The BYOM method maintains low operational overhead, demonstrating an average inference latency of just 4 milliseconds by utilizing efficient Gradient Boosted Trees for per-application predictions.
• The proposed cross-layer design is a generalizable paradigm, with potential applications extending beyond storage optimization to other critical system areas like resource management and network control.

About the Speaker(s)

The primary presenter for this work was Chenxi Yang, who conducted this research during an internship at Google. This project was a collaborative effort involving Google's storage analytics team and Google Deepmind. The extensive team of co-authors also included Yan Li, Martin Maas, Mustafa Uysal, Arif Merchant, and Richard McDougall. Chenxi Yang's ongoing research interests extend to applying similar small-model, cross-layer design principles to other system optimization challenges, such as network controlling, indicating a broader vision for practical ML integration in complex computing infrastructures.