Defusing the Kubernetes API Performance Minefield - Madhav Jivrajani & Marek Siarkowicz

Defusing the Kubernetes API Performance Minefield - Madhav Jivrajani & Marek Siarkowicz

Speakers: Madhav Jivrajani, Marek Siarkowicz

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=SdLLOcNZN5E

Overview

This talk, presented by Marek Siarkowicz, SIG lead of API Machinery and contributor to etcd, delves into the critical and often overlooked realm of Kubernetes API performance. While Kubernetes is celebrated for its ability to manage containerized workloads at scale, the underlying API server, particularly when interacting with Custom Resource Definitions (CRDs) and operators, harbors significant performance pitfalls. Madhav Jivrajani, unfortunately unable to attend, was also a key contributor to this work. The presentation aims to expose these "minefields," explain their root causes, and present recent and upcoming solutions designed to make the Kubernetes API server more robust and efficient.

The core problem addressed is the unexpected and often catastrophic memory consumption within the API server, primarily triggered by inefficient data retrieval patterns from etcd, Kubernetes' key-value store. This issue, frequently exacerbated by large CRD objects and misconfigured operators, can lead to control plane outages even in seemingly small clusters. Siarkowicz highlights that while Kubernetes itself is tested for massive scale (up to 5,000 nodes), these tests often don't cover the diverse and sometimes problematic usage patterns of custom resources. The talk is crucial for anyone operating Kubernetes at scale, developing operators, or building platforms on top of Kubernetes, as it provides both diagnostic understanding and practical mitigation strategies.

The presentation outlines a journey from identifying the performance "mines" to navigating them, and ultimately, to diffusing them through significant architectural and implementation improvements. It underscores that while much of Kubernetes' core functionality is mature, the API server's interaction with the broader ecosystem, especially concerning CRDs, has required substantial re-engineering to meet modern scalability demands. The insights shared are vital for ensuring the stability and cost-effectiveness of Kubernetes deployments, preventing the kind of memory spikes that can bring an entire control plane to its knees.

Background

▶ Watch: Introduction to Kubernetes API performance challenges (0:00)

Kubernetes, since version 1.6, has been rigorously tested for scalability, supporting clusters with up to 5,000 nodes, 150,000 pods, and 10,000 services. However, this impressive scalability often applies to Kubernetes' core components and resources. The landscape has evolved significantly, with CRDs becoming a ubiquitous mechanism for extending Kubernetes functionality. This widespread adoption has inadvertently created new performance challenges because many community-contributed CRDs and operators are not tested beyond small-scale deployments, leading to a gap between expected and actual performance.

A fundamental pattern in Kubernetes is the reconcile loop employed by controllers and operators. Unlike traditional applications that might fetch single rows of data for individual users, Kubernetes controllers often need to process the entire state of a cluster or a significant subset of it to make decisions. For instance, a controller might list 150,000 pods. When these resources, especially CRDs, contain large amounts of data (e.g., 500 MB per resource), this pattern becomes highly problematic. The speaker cited an incident where a seemingly small 50-node GKE cluster experienced a 20x memory spike in its API server, leading to a control plane collapse, all due to an operator upgrading and repeatedly listing large CRD objects.

The existing Kubernetes mechanisms designed to prevent resource abuse, such as limits and quotas, often fall short in these scenarios. While an individual object might be capped at 1 MB, a collection of objects (like a list of pods or CRDs) can easily accumulate to gigabytes in size. Furthermore, API Priority and Fairness (APF), which helps manage API server load, primarily accounts for CPU usage and does not effectively mitigate excessive memory allocations. This means that even if a controller is not hogging CPU, its repeated requests for large data sets can still exhaust API server memory, leading to Out-Of-Memory (OOM) errors and control plane instability. The core issue stems from the API server's internal handling of these large data requests, which involves multiple memory allocations and copies before the data even reaches the client.

Key Findings

▶ Watch: Real-world incident: operator upgrade spikes memory 20x (2:40)

The central discovery highlighted in the talk is the highly inefficient memory allocation pattern within the Kubernetes API server when handling large list requests, particularly those that bypass the internal cache. Marek Siarkowicz identified a "five allocation" problem for each such request, where a single gigabyte of data could trigger up to five gigabytes of memory allocation within the API server. This fundamental inefficiency was found to be the primary driver behind the unexpected memory spikes and control plane instability observed in production environments.

The existing caching mechanism, while effective for certain request types, was found to be underutilized by default configurations and specific API parameters. The speaker presented a "red table" illustrating that the majority of API list request combinations default to direct etcd access, bypassing the cache and incurring the heavy allocation penalties. This lack of cache utilization, combined with the inherent inefficiencies of JSON encoding for large datasets, created a significant performance minefield for operators and CRD developers.

To address these issues, the talk presented three key architectural and implementation improvements:

1. Consistent Reads from Cache (Kubernetes 1.31): This feature enables the API server's cache to serve "most recent" data requests consistently, avoiding direct etcd calls that were previously the default for such requests.
2. List from Snapshot (Kubernetes 1.33 Alpha): A re-implementation of the cache's storage layer using b-trees enables the API server to store historical snapshots of the cluster state in memory. This allows it to serve requests for specific resourceVersions directly from the cache, rather than relying on etcd's costly historical data retrieval.
3. Streaming Collection Encoding (Kubernetes 1.33 Beta): A custom re-implementation of JSON and Protobuf encoding was introduced to dramatically reduce memory allocations when serializing large lists of objects for client responses, moving away from monolithic in-memory blobs to a streaming approach.

These findings represent a concerted effort to shift the default behavior of the Kubernetes API server towards more cache-efficient and memory-optimized data handling, aiming to diffuse the performance "mines" that have plagued large-scale or CRD-heavy deployments.

Technical Deep Dive

▶ Watch: Kubernetes scalability tests often overlook CRD performance (4:00)

The core of the performance problem lies in the detailed mechanics of how the Kubernetes API server processes list requests, particularly when a request bypasses the internal cache and goes directly to etcd. Marek Siarkowicz meticulously outlined a "five allocation" problem for such requests:

1. Client Request to API Server: A client (e.g., an operator) sends a list request.
2. API Server to etcd: The API server translates this list request into a range query for etcd. etcd fetches the raw data from disk, deserializes it, and then serializes it into Protobuf format (as gRPC is used for etcd communication) before sending it back. This is the first significant allocation.
3. API Server Receives Data: The API server receives the Protobuf-encoded data from etcd, loading it into its own memory. This is the second allocation.
4. API Server Decodes Data: To process the data (e.g., for filtering or conversion), the API server must decode the Protobuf bytes into native Go objects. This involves a memory copy, marking the third allocation.
5. API Server Encodes Response: Finally, to send the response back to the client, the API server encodes the Go objects into the requested format (e.g., JSON or Protobuf). This is the fourth allocation.
6. Potential Filtering: While the API server can filter data in the storage layer (e.g., a Kubelet fetching only pods on its node), for most controller list requests, filtering is a pass-through, meaning the full dataset is processed.

This chain of operations means that fetching a 500 MB dataset could result in multiple gigabytes of memory usage within the API server. With multiple misbehaving clients or operators, these allocations quickly accumulate, leading to Out-Of-Memory (OOM) errors and control plane instability.

To mitigate this, Kubernetes employs an internal watch cache. When properly utilized, the cache can drastically reduce allocations:

• The cache maintains its own in-memory storage, holding a subset of etcd's state.
• It's populated by the API server requesting all data from etcd once, decoding it, and storing it in memory.
• Crucially, the cache maintains an open watch connection to etcd, receiving incremental updates (e.g., 1 MB for a single pod change instead of 500 MB for a full list). This means decoding only small updates at a time.
• When a client request hits the cache, the API server only needs to encode the already decoded data for the client, resulting in just one allocation.

However, the talk revealed that the cache's benefits were severely limited. A comprehensive table presented during the talk showed that out of all possible API list request arguments, only a handful actually leverage the cache efficiently. The default list request (without any parameters) was delegated directly to etcd, bypassing the cache entirely. This made it "very hard for most users to know what kind of performance they're getting."

To diffuse these performance mines, several significant improvements have been introduced:

1. Consistent Reads from Cache (Kubernetes 1.31):

• ResourceVersion Semantics: The concept of a resourceVersion is key, acting as a global logical clock. Clients can request data based on different resourceVersion semantics:
• Any: Any data available in the cache.
• Not older than: Wait until the cache is fresh enough, based on a minimum resourceVersion.
• Most recent: The latest state available.
• Exact: A specific historical resourceVersion.
• Continuation: For pagination, based on an exact resourceVersion with an offset.
• Solving "Most Recent": Previously, "most recent" requests went directly to etcd. Now, to serve this from the cache, the API server first queries etcd for its latest resourceVersion (a very cheap request). If the cache's resourceVersion is not up-to-date, the API server can "poke" the etcd watch connection. etcd, since version 3.3, supports watch progress notifications, allowing it to confirm the cache's freshness even if no new events occurred. This allows the cache to fulfill the "most recent" request, enabling the default list configuration to be served from the cache without hitting etcd. This feature is enabled by default in Kubernetes 1.31.

1. List from Snapshot (Kubernetes 1.33 Alpha):

• Addressing "Exact ResourceVersion": etcd can magically go back in time to serve historical resourceVersions, but the traditional API server cache could not. To enable this, a new re-implementation of the cache's storage was introduced in Kubernetes 1.32, based on b-trees.
• B-Tree Superpower: This new storage not only brought a 25% performance improvement and reduced allocations by 15% but also enabled a "superpower": the ability to snapshot and clone the cache.
• Snapshotting the Cache: Instead of updating the single cache storage and losing historical data, the API server can now clone the storage before applying a new watch event. This creates a snapshot of the cache's state at a specific resourceVersion. By doing this repeatedly, the API server can maintain a full history of all states in memory, ready to be served.
• Serving Exact Requests: If a client requests resourceVersion 42, the API server can directly access the corresponding snapshot. If the resourceVersion is too old (beyond the 75-second cleanup window for the watch cache) or not available, the request still delegates to etcd to maintain consistency. This feature is currently alpha in Kubernetes 1.33 and aims for beta and default enablement in 1.34. It also supports the "continuation" semantic.

1. Streaming Collection Encoding (Kubernetes 1.33 Beta):

• JSON Allocation Problem: Even when serving from the cache, encoding large JSON responses still incurred significant memory overhead (e.g., 1 GB payload requiring 2 GB of API server memory). This is because standard Go JSON encoders might load the entire list into memory as a single blob before writing it.
• Custom Implementation: A custom re-implementation of JSON and Protobuf encoding was introduced (via a KEP titled "Streaming Collection Encoding" in 1.33) that is now enabled by default.
• Streaming Objects: Instead of creating one huge in-memory blob, this custom encoder streams objects one by one. For example, when encoding a PodList, it writes the opening [ token, then each Pod object, and finally the closing ]. This drastically reduces peak memory allocations, as the entire list is never held as a single encoded byte array in memory. This is considered a temporary solution until Golang's JSON encoder version two provides native streaming capabilities.

These combined efforts represent a significant overhaul of the API server's data handling, moving towards a much more memory-efficient and cache-centric architecture, addressing critical scalability bottlenecks that were previously hidden "mines" for many users.

Demo / Proof of Concept

▶ Watch: Deep dive into 5 memory allocations per API list request (7:00)

While the talk did not feature a live, interactive demo, Marek Siarkowicz presented compelling performance results that served as a powerful proof of concept for the implemented improvements. These results, derived from official benchmarks, directly demonstrated the profound impact of the new features on API server memory usage.

The most striking benchmark result showed a dramatic reduction in memory consumption for a typical list operation. Before these optimizations, a list operation could consume up to 70 GB of memory within the API server. After the implementation of consistent reads from cache, list from snapshot, and especially streaming collection encoding, this memory usage plummeted to just 3 GB. This represents an over 95% reduction in peak memory allocation for list requests, effectively eliminating the risk of API server OOMs caused by large list responses.

This benchmark data provided concrete evidence that the identified "minefield" of performance issues could indeed be diffused. The speaker emphasized that these improvements would make Kubernetes "fully safe from memory perspective" for list operations once all features, including "List from Storage" (referring to List from Snapshot), are fully rolled out and enabled by default in upcoming releases like Kubernetes 1.34. The performance gains are a direct validation of the architectural changes and custom encoding strategies discussed, proving their effectiveness in real-world scenarios.

Defensive Implications

▶ Watch: How API server caching mitigates memory overhead (10:00)

The insights and solutions presented in this talk offer several critical defensive implications for Kubernetes operators, developers, and platform builders:

1. Understand API Usage Patterns: Defenders must recognize that not all Kubernetes API requests are created equal. Default list requests, especially before Kubernetes 1.31, often bypassed the cache and directly hit etcd, leading to severe memory implications. It's crucial to understand how operators and custom controllers are interacting with the API server, particularly regarding large data sets.
2. Read Documentation and Cloud Provider Recommendations: The speaker stressed the importance of reading Kubernetes documentation and cloud provider guidelines. These resources often recommend against large resource sizes and advocate for the List-Watch pattern and using Protobuf for reduced allocations. Adhering to these best practices, such as setting resourceVersion in requests, helps ensure cache utilization. GKE, for example, recommends limiting storage sizes.
3. Monitor Object Sizes: Actively monitor the size of CRD objects and other resources. Objects exceeding 1 MB, or collections of objects totaling gigabytes, are red flags. While Kubernetes has limits and quotas, they don't fully protect against the memory allocation issues discussed. Proactive monitoring can identify problematic resources before they cause an incident.
4. Run Scalability Tests: Relying solely on Kubernetes' internal scalability tests is insufficient, as they primarily focus on core components, not the diverse CRD ecosystem. Projects and users must run their own scalability tests in environments that mimic production dimensions. This helps validate performance before deployment or upgrades, catching regressions that could shift cache-hitting requests to direct etcd access.
5. Advocate for Open Source Project Scalability: The community should encourage and potentially fund open-source projects (especially those developing popular CRDs and operators) to define and meet their own scalability goals. This collective effort would ensure that common components are battle-tested for performance.
6. Leverage New Kubernetes Features:

• Kubernetes 1.31 (Consistent Reads from Cache): Ensure clusters are upgraded to at least 1.31 to benefit from the default list requests being served efficiently from the cache, significantly reducing etcd load and API server memory.
• Kubernetes 1.33 (Streaming Collection Encoding): This feature, enabled by default in 1.33, drastically cuts down memory allocations for encoding large JSON/Protobuf responses. Upgrading will immediately provide memory benefits, even for cache-served requests.
• Kubernetes 1.33 Alpha (List from Snapshot): For users needing to query specific historical resourceVersions, validate and enable this alpha feature. While not yet default, it offers substantial performance gains for such use cases by serving from the cache's b-tree-based snapshots. Prepare for its eventual beta and default enablement in Kubernetes 1.34.

1. Consider Alternatives for Extreme Scale: While Kubernetes extensions are powerful, for extremely high-scale or specialized use cases with massive data, consider whether an externalized custom API server might be more appropriate, though this introduces significant operational overhead. The speaker noted that most users prefer not to manage etcd or custom API servers, highlighting the importance of fixing CRD performance within the existing model.

By understanding these implications and actively adopting the recommended practices and new features, operators can significantly enhance the stability, performance, and memory efficiency of their Kubernetes control planes, effectively defusing the API performance minefield.

Key Takeaways

• CRDs and Operators Pose Unique Scalability Challenges: While Kubernetes core scales well, CRDs and operators often introduce performance pitfalls due to large object sizes and inefficient data retrieval patterns that are not covered by standard Kubernetes scalability tests.
• The "Five Allocation" Problem: Direct etcd access for large list requests leads to multiple memory allocations (deserialization, serialization, decoding, encoding), causing significant API server memory spikes and potential OOMs.
• Cache Utilization is Key, But Was Limited: The API server's internal watch cache drastically reduces allocations to one, but historically, most default and specific API requests bypassed it, leading to widespread performance issues.
• Recent Improvements Dramatically Boost Performance:
• Kubernetes 1.31 introduced Consistent Reads from Cache, making default "most recent" list requests cache-friendly.
• Kubernetes 1.33 brought Streaming Collection Encoding (beta, default) and List from Snapshot (alpha), which uses b-trees to store historical cache states, further reducing memory and enabling efficient historical queries.
• Proactive Measures are Essential: Operators must monitor CRD object sizes, run their own scalability tests, and adhere to best practices like the List-Watch pattern to prevent performance regressions.
• Significant Memory Reduction Achieved: Benchmarks show peak memory usage for list operations dropping from 70 GB to 3 GB, demonstrating the profound impact of these API server optimizations.

About the Speaker(s)

Marek Siarkowicz is a prominent contributor to the Kubernetes ecosystem, serving as the SIG lead of etcd and actively contributing to API Machinery. His work focuses on the core components that underpin Kubernetes' scalability and performance, particularly concerning the API server and its interaction with the etcd key-value store. His deep expertise in these areas makes him a leading voice in addressing complex performance challenges within large-scale Kubernetes deployments.

Madhav Jivrajani was a co-speaker for this talk but was unfortunately unable to attend. He has also contributed significantly to the understanding and improvement of Kubernetes API performance, having previously given talks on the intricate details of how the Kubernetes cache works. His work, alongside Marek's, is instrumental in identifying and resolving critical performance bottlenecks in the Kubernetes control plane.