The Missing Metrics: Measuring Memory Interference in Cloud Native Systems - Jonathan Perry

The Missing Metrics: Measuring Memory Interference in Cloud Native Systems

Speakers: Jonathan Perry

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=nXdGXdxmWNQ

Overview

In the increasingly complex landscape of cloud-native systems, maintaining high availability and cost efficiency while delivering superior user experience remains the "holy grail" for Site Reliability Engineers. This talk by Jonathan Perry, titled "The Missing Metrics: Measuring Memory Interference in Cloud Native Systems," addresses a critical, often invisible, threat to these goals: memory noisy neighbors. Perry argues that while engineers dedicate significant effort to optimize application performance, these gains can be obliterated by resource contention on shared server infrastructure, particularly related to memory caches and bandwidth. The presentation highlights that this problem, long understood and mitigated by hyperscalers, lacks an accessible, open-source solution for the broader Kubernetes community.

The core of Perry's presentation is a call to action for the Kubernetes community to adopt sophisticated measurement and mitigation techniques for memory interference. He details how applications sharing physical server resources can inadvertently degrade each other's performance by monopolizing crucial memory components, leading to dramatic increases in tail latencies (P95/P99). Drawing on extensive research and real-world case studies from industry giants, Perry introduces the Unvariance collector, an open-source project designed to provide the granular, real-time memory contention metrics necessary to identify and mitigate noisy neighbor issues, ultimately enabling higher system utilization and significant cost savings.

Background

▶ Watch: Introduction: The SRE's holy grail of performance (0:00)

The journey towards highly available and cost-efficient deployments in cloud-native environments is fraught with challenges, not least of which is the inherent complexity of distributed systems. As users demand richer functionality—better recommendations, advanced search, AI capabilities—applications become more feature-rich and intricate. This complexity often obscures the root causes of performance degradation, leading engineers to blame nebulous factors like "the spaghetti monster of complexity" or "the network." However, a significant portion of this degradation, Perry contends, stems from simple resource congestion on the underlying physical servers, for which visibility is critically lacking.

The concept of "noisy neighbors" is not new. For over a decade, well-known research universities and hyperscalers have published dozens of papers exploring resource allocation capabilities that allow for running 50% more transactions on the same servers and reducing tail latencies (P95 and P99) by factors of five to 14. These achievements, however, have largely remained within the walled gardens of these large organizations. The problem manifests when one application, the noisy neighbor, consumes a disproportionate share of a shared resource, thereby denying other applications access and degrading their performance. In this talk, Perry specifically focuses on the last-level caches (LLC), often referred to simply as caches, and memory bandwidth.

A cache noisy neighbor scenario unfolds when an application's working set, ideally residing in fast L1, L2, or L3 CPU caches, is evicted by another application's extensive use of the shared L3 cache. This forces the victim application to fetch data from slower DRAM, a process that can be up to 50 times slower than accessing L2 cache. In more extreme cases, where noisy neighbors run on adjacent hyperthreads, even L2 caches can be impacted. Google's experiments with synthetic noise generators on their production services, including web search, an online ML cluster, and an in-memory key-value store like Memcached, revealed dramatic slowdowns: up to five times for web search and ML clusters, and a staggering 14 times for Memcached at P95/P99 latencies.

The problem is exacerbated in multi-tenant cloud environments. While best practices often dictate separating big data analytics from latency-critical, user-facing applications, this isolation often breaks down when deploying on virtual machines (VMs) smaller than a full physical machine. In such scenarios, a user's interactive workload might run alongside a cloud provider's other tenants, potentially including batch analytics jobs from "some random dude on the internet." Perry notes a lack of evidence that cloud providers offer substantial protection against memory noisy neighbors in these shared environments. This means that months or years of optimization work—adding database indices, refining data schemas, breaking down services for parallel execution—can be "obliterated" by an adjacent noisy neighbor.

Furthermore, noisy neighbors aren't exclusively external. Internal workloads, such as garbage collection (GC), can also cause significant interference. Researchers from MIT demonstrated this by running Memcached alongside a garbage-collected workload. During the memory-intensive "mark phase" of the GC, memory bandwidth saturated, causing Memcached's 99.9th percentile latency to jump from a stable 50 microseconds to three orders of magnitude higher (a thousand times increase). This illustrates that even well-intentioned, internal processes can become noisy neighbors, underscoring the pervasive nature of the problem.

Key Findings

▶ Watch: The hidden capability for 50% more transactions (3:00)

The talk reveals several critical insights into the nature and impact of memory interference in cloud-native systems, alongside the strategies hyperscalers have employed to overcome these challenges.

Firstly, despite widespread efforts to optimize cloud deployments, CPU utilization remains strikingly low for many organizations. DataDog surveys and reports from major companies indicate average cluster utilization often falls between 10% and 40%. This inefficiency is directly linked to the need to scale out systems to avoid performance degradation caused by memory contention, effectively operating at lower loads to maintain acceptable P95/P99 latencies. Perry critically labels this "an anti-pattern" that is both wasteful and inefficient.

Secondly, hyperscalers like Google have demonstrated that significant efficiency gains are achievable. Google, for example, increased its cluster utilization from around 35% in 2011 to 50% in 2019, representing a 1.5x increase in efficiency. Perry attributes these gains primarily to two factors: advancements in vertical autoscaling (ensuring CPU and memory requests fit the workload) and, crucially, handling noisy neighbors, which reduces tail latencies and enables higher workload density. This proves that mitigating memory interference is a key enabler for substantial infrastructure savings and performance improvements.

Thirdly, traditional methods of performance monitoring are inadequate for detecting and mitigating memory noisy neighbors. Measuring P95/P99 latency is too noisy and slow to react, requiring hundreds of transactions for a reliable signal. While Cycles Per Instruction (CPI) can indicate memory contention (higher CPI means more CPU waiting for memory), it's also noisy, reflects other system activities, and requires complex application-specific profiling to determine "good" values. Google has deployed CPI-based systems across all its shared clusters since 2013, indicating its viability but also its complexity.

Finally, the talk emphasizes that direct measurement of memory contention is the most effective approach. By directly monitoring the utilization of memory bandwidth and caches, systems can identify applications consuming more than their fair share and limit them. Alibaba Cloud, for instance, has successfully deployed a production system based on direct memory contention event collection for over two years, operating on an order of a million cores as of 2020. The critical requirement for this approach is very frequent measurements, specifically at a 1-millisecond granularity, to capture ephemeral contention events that would otherwise be missed by coarser sampling intervals. This finding forms the foundation for the open-source Unvariance collector project.

Technical Deep Dive

▶ Watch: How cache noisy neighbor evicts your application data (5:45)

Addressing memory noisy neighbors requires both precise measurement and effective mitigation mechanisms. Perry delves into the hardware and software capabilities that make this possible, as well as the architecture of the proposed open-source solution.

At the heart of the problem are the shared resources within a physical server: the CPU's last-level caches (LLC) (L1, L2, L3) and memory bandwidth. When a noisy neighbor extensively uses the shared L3 cache, it evicts other applications' data, forcing them to retrieve it from much slower DRAM. In extreme cases, L2 cache eviction can occur. Similarly, saturation of memory bandwidth by one workload can drastically increase memory access latencies for all co-located applications. For example, memory bandwidth saturation can increase latency from 75 nanoseconds to 250 nanoseconds (a 3-4x increase), while cache evictions can lead to 10x slowdowns, and often these effects compound.

Modern CPUs provide mechanisms to control and monitor these shared resources. While specific CPU extensions like Intel's Resource Director Technology (RDT) or AMD's equivalent Performance Monitoring Unit (PMU) extensions are not explicitly named, the talk implies their existence. These hardware capabilities allow the operating system to dictate how much cache and memory bandwidth each application can consume. Even without direct hardware support for fine-grained control, basic mitigation can be achieved by pinning noisy neighbors to a small number of cores and potentially reducing their frequency to limit their ability to cause damage.

Crucially, standard container isolation mechanisms, such as those provided by Docker or Kubernetes, are not inherently designed to isolate performance at this granular level. Instead, the necessary controls are exposed through a different subsystem: resource control (resctrl), configurable via the sysfs interface in Linux. This provides the foundational software layer for interacting with the CPU's resource allocation features.

The Unvariance collector, an Apache 2 licensed project, is designed to leverage these mechanisms. Its core technical approach revolves around direct, high-frequency measurement of memory contention.

The collector's architecture involves several key components:

1. RMID Assignment: Whenever a new thread or task (e.g., a container process) is forked, the collector assigns a unique Resource Monitoring ID (RMID) to it. This RMID is a hardware-level identifier.
2. Context Switch Monitoring: On every context switch between applications, the collector informs the CPU which RMID is now active. This effectively "colors the traffic," allowing the CPU to associate memory accesses and cache usage with specific workloads.
3. 1ms Granularity Sampling: Every 1 millisecond, the collector queries the CPU for the amount of memory accesses and cache utilization associated with each active RMID (i.e., each "color"). The hardware supports hundreds of these RMIDs, sufficient for numerous containers on a system.
4. Shared Memory Buffer: All this high-frequency telemetry, along with RMID-to-container allocations, flows into a shared memory buffer.
5. User Space Analysis: A user-space component then reads and analyzes this data. The current goal is to output raw telemetry into Parquet files, a columnar, compressed format. This raw data will be used to develop and back-test advanced detection algorithms to identify noisy neighbors. The long-term vision is to output observability statistics (e.g., "this pod was a noisy neighbor 1% of the time") and use these detections to automatically mitigate noisy neighbor behavior by configuring resctrl limits.

A significant technical challenge in achieving 1ms granularity is jitter in high-resolution timers. Perry demonstrated that while workable, smaller VMs (e.g., four-core slices) can exhibit significantly more noise and jitter (up to 300 microseconds, or 30% of the measurement interval) compared to larger, bare-metal instances (e.g., 96 cores). This jitter means that different cores might respond to timers at slightly different intervals, complicating the aggregation of measurements. The collector's design has to account for this variability.

Regarding overhead, the Unvariance collector is designed for a target of 0.1% overhead for in-line measurement operations and 1% for user-space analysis. Perry argues that even a higher overhead (e.g., 5% average service time increase) would be a worthwhile trade-off if it significantly reduces tail latencies (e.g., P95 from 250ms to 75ms), underscoring the severe impact of unmitigated noisy neighbor issues.

Demo / Proof of Concept

▶ Watch: Google's production services experience up to 14x slowdowns (6:30)

While the talk thoroughly describes the architecture and capabilities of the Unvariance collector, it does not feature a live demonstration or a specific, isolated proof-of-concept for a particular noisy neighbor scenario. Instead, the focus is on the design of the collector and the methodology for collecting the necessary high-granularity memory contention metrics.

Perry outlines the project's current status and future plans, emphasizing the need for real-world data to refine detection algorithms. The collector's ability to output raw telemetry into Parquet files is crucial for this. He explicitly calls for contributions of production data, perhaps from test or staging clusters with real traffic, to help build better noisy neighbor detectors. This collaborative approach, rather than a singular demo, forms the "proof of concept" in the context of the talk: demonstrating the feasibility of collecting the missing metrics and inviting the community to help build the solution.

Defensive Implications

▶ Watch: Why optimization efforts are undone by interference (9:00)

The insights and proposed solutions from this talk carry significant implications for how organizations design, deploy, and manage their cloud-native infrastructure, especially within Kubernetes environments.

1. Re-evaluate VM Sizing and Deployment Strategy: For organizations running a thousand cores or more, Perry strongly recommends preferring fewer, larger bare-metal instances (e.g., 100-core machines) over many smaller VMs (e.g., four or eight-core machines). This strategy reduces the likelihood of unknown, potentially aggressive batch workloads from other tenants becoming noisy neighbors, allowing for better control over co-located workloads.
2. Adopt Direct Memory Contention Monitoring: The talk highlights the inadequacy of traditional latency (P95/P99) and CPU efficiency (CPI) metrics for real-time noisy neighbor detection. Defenders should integrate tools like the Unvariance collector to gain direct, 1-millisecond granularity visibility into cache utilization and memory bandwidth consumption per application. This provides the essential signal to identify and attribute performance degradation to memory interference.
3. Shift from Scaling Out to Mitigating: The current "anti-pattern" of scaling out systems (adding more instances) to reduce load and avoid high P95/P99 latencies is wasteful. Instead, organizations should focus on implementing noisy neighbor mitigation. By preventing memory contention from degrading performance, systems can run at higher CPU utilization and density without breaching SLOs, leading to substantial cost savings.
4. Implement Automatic Mitigation Strategies: The ultimate goal is to move beyond mere detection to automatic mitigation. By leveraging the resctrl mechanisms in Linux, identified noisy applications can have their cache and memory bandwidth usage restricted in real-time. This can be particularly effective for transient noisy events like garbage collection. Even if full automation isn't immediately feasible, the collected data can inform anti-affinity rules to prevent consistently clashing workloads from being co-located.
5. Enable Product Innovation: By mitigating noisy neighbors and ensuring consistent performance, product teams can focus on building more features without the constant pressure to optimize existing systems to meet SLOs. This allows for faster innovation and improved user experience.
6. Understand Cloud Provider Limitations: Be aware that cloud providers may offer little protection against memory noisy neighbors on smaller, shared VM instances. Organizations must take proactive steps to monitor and mitigate these issues themselves, rather than assuming platform-level isolation.

Key Takeaways

• Memory noisy neighbors, arising from contention over shared CPU caches and memory bandwidth, are a significant, often invisible, cause of performance degradation (especially tail latencies like P95/P99) and increased operational costs in cloud-native systems.
• Traditional container isolation mechanisms (e.g., Kubernetes) do not effectively prevent performance interference from memory noisy neighbors, leading to widespread underutilization of CPU resources (typically 10-40%).
• Hyperscalers have achieved substantial efficiency gains (e.g., Google increasing utilization from 35% to 50%) by developing internal systems for memory noisy neighbor detection and mitigation, alongside advancements in vertical autoscaling.
• Effective detection requires direct, high-frequency measurement of memory contention events (1-millisecond granularity) using hardware-level mechanisms exposed via Linux resctrl, as opposed to noisy and slow-reacting P95/P99 latency or complex Cycles Per Instruction (CPI) metrics.
• The Unvariance collector is an open-source (Apache 2) project aiming to bring these advanced measurement capabilities to the Kubernetes community, providing 1ms granularity telemetry on cache and memory bandwidth utilization for detection and future automatic mitigation.
• Mitigating memory noisy neighbors enables higher system density and CPU utilization, reduces infrastructure costs, improves application performance, and frees product teams to focus on feature development rather than constant performance optimization.

About the Speaker(s)

Jonathan Perry is a recognized expert in the field of resource contention and performance in distributed systems. He holds a PhD specializing in network noisy neighbor mitigation, demonstrating a deep academic and practical understanding of resource interference. His entrepreneurial journey includes founding a network observability company, which was subsequently acquired by Splunk. Furthermore, Jonathan is a maintainer on the OpenTelemetry network collector, showcasing his commitment to open-source contributions in the observability space. Currently, his work is focused on extending these efforts to address memory noisy neighbor mitigation, bringing his extensive experience to this critical area of cloud-native performance.