A Huge Cluster or Multi-Clusters? Identifying the Bottleneck - Paco Xu & Saiyam Pathak

A Huge Cluster or Multi-Clusters? Identifying the Bottleneck

Speakers: Paco Xu, Kubernetes Steering Committee Member, Kube Maintainer, DaoCloud; Saiyam Pathak, Principal Developer Advocate, Loft Labs

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=6l5zCt5QsdY

Overview

In this insightful KubeCon EU talk, Paco Xu and Saiyam Pathak delve into the perennial question facing Kubernetes architects: should organizations opt for a single, massive Kubernetes cluster or distribute workloads across multiple smaller clusters? Titled "A Huge Cluster or Multi-Clusters? Identifying the Bottleneck," the presentation meticulously explores the scaling limits of Kubernetes, identifies common performance bottlenecks, and offers a comprehensive array of solutions ranging from control plane optimizations to advanced multi-tenancy strategies.

The speakers, both prominent figures in the Kubernetes community, address a critical challenge for enterprises navigating their cloud-native journey. As Kubernetes adoption matures, the demand for larger clusters to consolidate resources and simplify management, or for numerous smaller clusters to enhance isolation and flexibility, becomes paramount. This talk provides a detailed technical analysis of the trade-offs involved, equipping practitioners with the knowledge to make informed decisions for their specific operational requirements and organizational scales.

Background

▶ Watch: Audience survey: Prevalence of large and multi-clusters (1:58)

The journey of Kubernetes scaling has been a continuous evolution, pushing the boundaries of what a single cluster can manage. Historically, the official Kubernetes documentation recommended a maximum of 5,000 nodes, a number tested by SIG Scalability. However, real-world deployments have consistently surpassed this benchmark. Early public reports from companies like OpenAI in 2018 demonstrated clusters scaling to 2,500 nodes. More recently, organizations such as ByteDance, Alibaba Group, and OpenAI have reported clusters ranging from 10,000 to 15,000 nodes. A significant milestone was announced at KubeCon North America, where TKE showcased scaling to an impressive 65,000 nodes.

Despite these advancements, operating large-scale Kubernetes clusters introduces a unique set of challenges and bottlenecks. A Reddit poll conducted by the speakers highlighted common pain points reported by the community, including issues with events, autoscaling mechanisms (like cluster-autoscaler), DNS resolution, IP tables performance, and endpoint sync. A crucial insight from this feedback was that node number alone is an insufficient metric for cluster size; the total object number within the cluster often provides a more accurate representation of its operational complexity and pressure. Additionally, the need for robust multi-tenancy solutions was frequently cited as a significant concern for large clusters.

General bottlenecks in huge clusters often center around the control plane. The ETCD key-value store, serving as Kubernetes' primary datastore, is a frequent bottleneck due to its write-heavy nature. DNS and storage challenges also emerge as critical issues at scale, alongside the complexities of node management and monitoring.

The motivations for building huge clusters are compelling: reduced maintenance costs, centralized management, and easier policy enforcement. Such clusters also offer resilience against single rack failures if designed across multiple availability zones. However, the drawbacks are equally significant. Native Kubernetes namespaces provide limited isolation for true multi-tenancy, leading to a potentially large blast radius in case of failures or security incidents. Furthermore, large clusters can restrict end-user flexibility, particularly concerning Kubernetes version selection (where nodes must be within N-3 versions of the control plane) and operator version dependencies. The community's current long-term support (LTS) for Kubernetes versions, typically around one year, further compounds version management challenges for monolithic clusters. These issues underscore the trade-off between the perceived simplicity of a single large cluster and the operational complexities it introduces.

Key Findings

▶ Watch: Reddit community feedback on common cluster bottlenecks (4:05)

The talk presents several key findings and contributions to the understanding of Kubernetes scalability and multi-tenancy:

1. Kubernetes Can Scale Significantly Beyond Official Benchmarks: While 5,000 nodes was a long-standing recommendation, real-world deployments have pushed this limit dramatically, with clusters reaching up to 65,000 nodes. The community is actively working on features to support even larger scales, with ETCD sharding potentially enabling clusters up to 30,000 nodes.
2. ETCD Remains the Primary Bottleneck: Despite continuous improvements, ETCD is consistently identified as the core bottleneck in large Kubernetes clusters due to the sheer volume of reads and writes. Solutions heavily focus on reducing pressure on ETCD or finding alternative datastores.
3. The "Huge Cluster vs. Multi-Cluster" Dilemma is Nuanced: There are clear benefits and drawbacks to both approaches. Huge clusters offer centralized management and reduced overhead, but at the cost of blast radius, isolation, and flexibility. Multi-cluster strategies, while adding management complexity, provide superior isolation, version flexibility, and a smaller blast radius. The choice depends heavily on an organization's specific requirements for tenancy, isolation, and operational overhead.
4. Multi-Tenancy Solutions are Evolving Rapidly: The Kubernetes ecosystem is developing a rich spectrum of multi-tenancy solutions, moving beyond simple namespaces to more robust models like "Namespace as a Service," "Kubernetes API as a Service," and "Control Plane as a Service" (both internal and external). These solutions aim to provide stronger isolation and flexibility without the full overhead of dedicated physical clusters.
5. Community Efforts are Continuously Improving Scalability: Ongoing Kubernetes Enhancement Proposals (KEPs) and new features, such as ETCD server overrides for sharding, node heartbeats, watch bookmarks, endpoint slices, and the upcoming NF tables GA in Kubernetes 1.33, demonstrate a sustained commitment to addressing scalability challenges within the core project.
6. Proactive Performance Testing is Crucial: Tools like Quark are essential for simulating large-scale clusters and testing control plane components, schedulers, and custom operators before deploying to production, mitigating risks associated with scale.

Technical Deep Dive

▶ Watch: Advantages and disadvantages of using huge clusters (5:10)

The technical core of the talk revolves around identifying specific bottlenecks and detailing an array of solutions for both huge clusters and multi-cluster environments.

Addressing Huge Cluster Bottlenecks

1. API Server Pressure:

• Rate Limiting and APF (API Priority and Fairness): These mechanisms help manage the load on the API server, ensuring critical operations are prioritized.
• Caching: Implementing read caches can significantly reduce the number of direct calls to the API server, especially for frequently accessed data. DaemonSets that make numerous API calls are identified as expensive and should be optimized or avoided where possible.
• Validation Policies: Tools like OPA Gatekeeper or Kyverno can reduce ETCD pressure by validating resources before they are persisted, preventing invalid or excessive objects from reaching the datastore.

1. Scheduler Optimization:

• Choosing a proper scheduler tailored to specific workloads and performing scenario-based testing are crucial for maintaining performance in large clusters.
• Tools like Quark are invaluable for testing scheduler performance under simulated large-scale conditions.

1. ETCD Management and Alternatives:

• ETCD Tuning: Optimizing disk I/O, network latency, and heartbeat intervals are fundamental. The official ETCD tuning page provides detailed recommendations.
• ETCD Server Overrides for Sharding: This feature (detailed in specific KEPs) allows different types of objects (e.g., events, leases, or custom API groups) to be stored in separate ETCD clusters. This sharding significantly reduces the load on any single ETCD instance, potentially enabling clusters up to 30,000 nodes. For example, Pod objects, being some of the largest and most numerous, can also be sharded across multiple ETCD clusters.
• ETCD Replacements:
• KruiseBrain (from ByteDance): An open-source project used to scale clusters to 20,000 nodes. It currently has limitations, supporting only Kubernetes 1.x, and requires a special, not-yet-open-sourced patch for optimal performance.
• Kine: Often used in edge scenarios, Kine provides a pluggable backend for ETCD, allowing Kubernetes to use different databases (like MySQL, PostgreSQL, SQLite, or S3) as its datastore. While promising for certain use cases, its suitability for massive general-purpose clusters requires thorough testing.
• ETCD 3.5 Improvements: Version 3.5 of ETCD introduced significant performance enhancements. It's critical to use patch versions after 3.5.6 due to critical issues in earlier releases.
• Google's Spanner: Google has replaced ETCD with Spanner in some internal large-scale Kubernetes deployments, demonstrating the potential for highly distributed, globally consistent databases to handle extreme scale.

1. Network and DNS:

• CNI (Container Network Interface): The choice and configuration of the CNI play a vital role in network performance.
• DNS Optimization: Implementing DNS caching, scaling DNS servers, or using alternatives like host alias can mitigate DNS-related bottlenecks.
• NF Tables: The upcoming NF tables implementation, reaching General Availability (GA) in Kubernetes 1.33, promises improved performance over traditional IP tables for kube-proxy.
• Pod IP Range and Node IPs: Careful planning of IP ranges and network topology is essential for cross-room or multi-subnet deployments, leveraging concepts like affinity.

1. Autoscaling and Storage:

• Carpenter: A fast and cost-effective node autoscaler that integrates with the Kubernetes community.
• KEDA (Kubernetes Event-Driven Autoscaling): Extends HPA capabilities for event-driven workloads.
• Storage: A complex topic on its own, efficient CSI (Container Storage Interface) drivers and robust storage backends are critical for performance and reliability at scale.

Kubernetes Community Updates

Recent Kubernetes releases have brought significant improvements for large clusters:

• KEPs (Kubernetes Enhancement Proposals): Several KEPs focus on scalability, including those related to ETCD server overrides for sharding, node heartbeats (improving node status reporting), watch bookmark (optimizing watch operations), and endpoint slice (reducing the size of endpoint objects).
• Kubernetes 1.33: This release includes many changes related to caching, streaming, and the GA of kube-proxy NF tables, all contributing to better performance and scalability.

Multi-Tenancy and Multi-Cluster Solutions

The talk presents a spectrum of multi-tenancy approaches:

1. Namespaces: The most basic form, offering limited isolation, especially for cluster-scoped resources like CRDs.
2. Namespace as a Service: Groups namespace constructs (resource quotas, limit ranges, network policies) for intelligent provisioning. Projects like Capsule and Hierarchical Namespaces fall into this category, though maintenance can be a concern.
3. Kubernetes API as a Service: Uses a proxy in front of the API server to filter what users can see, including CRDs. Capsule Proxy and KCP are examples, providing more granular control over resource visibility for different teams.
4. Dedicated Clusters: Provides the strongest isolation, with each user or team getting their own Kubernetes cluster. Projects like Karmada (a multi-cluster management system) and KubeSlice facilitate this, often with a central control plane managing agents on individual clusters. This is expensive but offers unparalleled isolation.
5. Control Plane as a Service (CPaaS):

• Internal CPaaS: Runs control planes as pods within an existing host cluster, syncing resources to the host.
• External CPaaS: A management control plane creates control plane pods, which then connect to worker nodes (running kubelet) on separate virtual machines. This model, exemplified by k0s, Hypershift, and vCluster, is suitable for providing Kubernetes as a service.

vCluster as a Specific Solution

vCluster is highlighted as an open-source project that creates virtual Kubernetes clusters on top of a single host Kubernetes cluster. Each vCluster runs its control plane as a StatefulSet on the host. This approach offers:

• Strong Tenant Isolation: Each team gets a kubeconfig file for their virtual cluster, isolating their workloads and views.
• Version Flexibility: Different vClusters can run different Kubernetes versions (e.g., 1.31, 1.32, 1.33) and different versions of operators/CRDs, enabling independent upgrades and testing.
• Reduced Blast Radius: A failure in one virtual cluster does not impact others.
• Resource Sharing: Essential host-level components like Ingress controllers, Cert-Manager, and Falco can be shared across multiple vClusters, reducing resource duplication and operational overhead compared to dedicated clusters.
• Syncer Component: A syncer continuously copies resources from the virtual cluster's control plane to the host cluster, where the actual scheduling and execution take place. This ensures that the virtual cluster appears as a full-fledged Kubernetes environment to its users while leveraging the host's underlying infrastructure.

Demo / Proof of Concept

▶ Watch: Solutions for API server pressure and scheduler optimization (8:00)

Saiyam Pathak provided a concise and effective live demonstration of vCluster to illustrate its capabilities. The demo began by showing a three-node host Kubernetes cluster using kubectl get nodes.

The core of the demo involved creating a new virtual cluster with a single command: vcluster create demoert -f vcluster.yaml. This command initiated the provisioning of a new vCluster. The speaker then showed kubectl get pod -A on the host cluster, revealing a StatefulSet pod coming up, which represents the control plane of the newly created virtual cluster. This visually confirmed that the virtual cluster's control plane runs as a standard Kubernetes workload on the host.

Once the vCluster was ready, the demo automatically switched the kubectl context to the new virtual cluster. This immediately demonstrated the isolation benefit: running kubectl get pod -A in the new context showed only the networking pod within the virtual cluster itself, not the underlying host cluster pods. This simulated the experience of a tenant who only sees their dedicated environment.

Further showcasing the flexibility and shared resource model, the speaker then deployed an application along with an issuer certificate and an ingress resource within the virtual cluster context. Critically, the speaker emphasized that the ingress controller and Cert-Manager were not installed within the virtual cluster; instead, the vCluster was configured to sync these resources to the host cluster, leveraging the host's existing ingress and certificate management infrastructure. This highlights vCluster's ability to provide a full Kubernetes experience to tenants while optimizing resource utilization by sharing expensive cluster-level components.

The demo concluded by accessing the deployed application via the ingress URL, which successfully displayed a "Welcome to KubeCon 2025" message, confirming the end-to-end functionality of the virtual cluster and its integration with host-level services. This demonstration effectively illustrated how vCluster addresses challenges related to isolation, versioning, and resource sharing in multi-tenant environments.

Defensive Implications

▶ Watch: Mitigating network, CNI, DNS, and IP tables bottlenecks (9:40)

For organizations operating or planning to operate Kubernetes at scale, the insights from this talk carry significant defensive implications:

1. Proactive Performance Testing is Non-Negotiable: Before scaling to production, extensive performance testing using tools like Quark or Group-Mark is crucial. This allows identification and mitigation of bottlenecks related to the API server, scheduler, ETCD, and network under simulated extreme loads. Understanding how custom operators and applications behave at scale is equally important.
2. Strategic Choice of Cluster Architecture: Defenders must carefully evaluate the trade-offs between huge monolithic clusters and multi-cluster strategies based on their specific security, compliance, and operational requirements.

• Blast Radius: Multi-cluster or multi-tenancy solutions (like vCluster) significantly reduce the blast radius compared to a single huge cluster. A compromise or failure in one tenant's virtual cluster is isolated, preventing lateral movement or widespread impact across the entire organization.
• Isolation: For environments requiring strong tenant isolation (e.g., regulated industries, distinct business units), solutions like "Control Plane as a Service" or dedicated clusters are preferable over basic namespaces.
• Version Drift and Patching: Multi-tenancy solutions that allow independent Kubernetes versioning per tenant can simplify patching and upgrades, as defenders can roll out updates gradually without impacting all workloads simultaneously.

1. ETCD Hardening and Optimization: Given ETCD's critical role and frequent identification as a bottleneck, defenders must prioritize its security and performance. This includes:

• Implementing ETCD server overrides for sharding to distribute load and potentially isolate sensitive data.
• Following official tuning guidelines for disk, network, and heartbeat settings.
• Exploring ETCD replacements (like KruiseBrain or Kine) or alternatives (like Spanner) if extreme scale demands it, but with thorough security vetting.
• Ensuring ETCD is running on recommended patch versions (e.g., ETCD 3.5.6+).

1. API Server Protection: Leverage API Priority and Fairness (APF) and rate limiting to protect the API server from abusive or resource-intensive requests. Implement validation policies (e.g., OPA Gatekeeper, Kyverno) to prevent malformed or excessively large objects from burdening the control plane.
2. Network and DNS Resilience: Optimize CNI performance, implement robust DNS caching strategies, and prepare for the adoption of NF tables in kube-proxy for improved network performance and security.
3. Continuous Monitoring and Alerting: Comprehensive monitoring of control plane components (API server, scheduler, controller-manager, ETCD) and data plane components (Kubelets, CNIs) is essential to detect performance degradation or anomalous behavior indicative of scaling issues or attacks.

By adopting a proactive, multi-faceted approach to performance, security, and architecture, defenders can ensure their Kubernetes deployments remain robust, scalable, and resilient against the challenges of massive scale.

Key Takeaways

• Kubernetes can scale far beyond traditional benchmarks, with real-world deployments reaching 65,000 nodes, but this introduces significant operational complexity.
• ETCD remains the primary bottleneck for large clusters, necessitating advanced tuning, sharding with ETCD server overrides, or exploring alternative datastores.
• The choice between a single huge cluster and multiple smaller clusters depends on factors like isolation needs, blast radius concerns, and version flexibility requirements.
• A spectrum of multi-tenancy solutions, from enhanced namespaces to "Control Plane as a Service" (like vCluster), provides stronger isolation and flexibility than basic Kubernetes namespaces.
• vCluster offers a compelling model for multi-tenancy by running virtual control planes on a host cluster, enabling independent versioning and shared host-level components.
• Ongoing Kubernetes community updates, including KEPs for ETCD sharding, node heartbeats, and NF tables, continuously improve scalability and performance.
• Proactive performance testing with tools like Quark is essential to validate cluster behavior and identify bottlenecks before production deployment.

About the Speaker(s)

Paco Xu is a distinguished member of the Kubernetes community, serving on the Kubernetes Steering Committee and as a Kube Maintainer. He works at DaoCloud, a cloud-native company, bringing deep expertise in Kubernetes core development and large-scale cluster management. His contributions are instrumental in shaping the future of Kubernetes scalability.

Saiyam Pathak is a Principal Developer Advocate at Loft Labs, known for creating vCluster. He is also the founder of CubeSimplify and BuildSafe, and identifies as a "KubeStronaut." Saiyam is highly active on social platforms and is a vocal proponent of virtual clusters and simplifying Kubernetes operations for developers and platform teams.