Beyond the Limits: Scaling Kubernetes Controllers Horizontally - Tim Ebert, STACKIT

Beyond the Limits: Scaling Kubernetes Controllers Horizontally

Speakers: Tim Ebert, STACKIT

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=OTzd9eTtLRA

Overview

In the realm of Kubernetes, controllers and operators are fundamental to achieving the desired declarative state management. They are the workhorses that observe, react, and enforce the configurations defined by users. However, a long-standing challenge for these critical components has been their inherent limitation in horizontal scalability. Due to the need to prevent conflicting reconciliations, most Kubernetes controllers rely on leader election, an active-passive High Availability (HA) mechanism that designates only one instance as the active leader at any given time, preventing true scale-out.

This talk, delivered by Tim Ebert of STACKIT, delves into this crucial limitation and presents a meticulously designed, open-source solution for achieving horizontal scalability in Kubernetes controllers. Ebert, whose team at STACKIT manages thousands of Kubernetes clusters using the Gardener project, faces the challenge of running controllers at immense scale daily. His master's thesis, "Horizontally Scalable Kubernetes Controllers," forms the basis of this innovative approach, which draws inspiration from distributed database sharding techniques.

The article will explore the core problem, the proposed architecture, its technical underpinnings, and practical implementation details, demonstrating how controllers can move beyond the confines of single-instance operation to handle significantly higher loads and object churn. This work is particularly relevant for platform providers, large enterprises, and anyone building complex, high-performance operators that need to manage a vast number of custom resources or experience high rates of change.

Background

▶ Watch: Introduction to Kubernetes controllers and scalability challenge (0:00)

Kubernetes controllers are the core automation engines of the platform, continuously working to reconcile the current state of the cluster with the desired state declared in API objects. Their operational cycle typically involves several key steps:

1. Watching: Controllers monitor the Kubernetes API server for changes to specific API objects.
2. Caching: Upon receiving change events, they cache these objects in memory for fast retrieval, reducing load on the API server.
3. Enqueuing: If relevant changes are detected, the affected object is enqueued for later reconciliation.
4. Reconciling: The controller reads the object from its cache and, if necessary, performs actions to bring the cluster's state in line with the object's desired state (e.g., creating Pods, Deployments, Services).
5. Reporting Status: Finally, the controller updates the object's status section or records Kubernetes events to reflect its observed state and progress.

The critical hurdle to horizontal scalability stems from the need to prevent conflicting reconciliations. If multiple instances of a controller were to simultaneously attempt to modify the same object, race conditions, inconsistent states, and unpredictable behavior would ensue. To circumvent this, Kubernetes controllers traditionally employ leader election, a mechanism that ensures only one instance of a controller is active and responsible for making changes at any given moment. This is typically achieved using a Lease resource in a designated namespace, where the active leader's identity is recorded.

Tim Ebert demonstrated this limitation with a "web hosting operator" example. When a single instance of the operator is running, it successfully reconciles Website objects, creating associated deployments, ingresses, and services. However, when the operator deployment is scaled to multiple replicas, only one instance assumes the leadership role, as indicated by the Lease object. The other instances remain in a passive, "pre-warmed" state. While this provides an active-passive HA setup—allowing for quick failover if the leader instance fails—it offers no performance improvement or horizontal scaling for increased throughput. The system's capacity remains limited by the single active instance.

For environments like STACKIT's Kubernetes Engine, which manages thousands of customer clusters, this limitation becomes a significant bottleneck. Running controllers at such a scale, where a single controller might be responsible for hundreds of thousands of objects or experience thousands of changes per second, necessitates a more robust scaling mechanism than traditional leader election can provide. The absence of a standard, horizontally scalable solution for Kubernetes controllers has long been a challenge for those operating at the frontier of Kubernetes adoption.

Key Findings

▶ Watch: Why leader election limits horizontal controller scalability (4:45)

The central finding and contribution of Tim Ebert's work is a novel design for horizontally scaling Kubernetes controllers by applying sharding mechanisms commonly found in distributed databases. This approach allows multiple instances of a controller to actively reconcile different subsets of API objects simultaneously, thereby increasing the overall capacity and throughput of the system.

The core principles underpinning this design are:

• Dynamic Membership and Failure Detection: Inspired by systems like Bigtable, the design incorporates a mechanism to dynamically track which controller instances (referred to as shards) are available and healthy within a "ring" of controllers.
• Automatic Failover: If a shard goes down, the objects it was responsible for are automatically reassigned to other available shards, ensuring continuous reconciliation.
• Rebalancing: When new shards join the ring, the distribution of objects is rebalanced across all available instances, optimizing resource utilization and performance.
• Label-Based Mechanism: The entire system leverages native Kubernetes API machinery, primarily using labels on API objects to assign them to specific controller instances. This avoids the need for complex, external coordination services.
• Prevention of Concurrent Reconciliations: Crucially, even with multiple active instances, the design guarantees that any single API object is always reconciled by only one controller instance at a time, preventing conflicts.
• Reusable Implementation: The design is implemented as a reusable library, allowing developers to easily integrate horizontal scalability into their existing or new controller-runtime based Kubernetes controllers with minimal code changes.

Load testing experiments conducted as part of the master's thesis demonstrated the efficacy of this sharding approach. The overhead introduced by the sharding infrastructure (the Sharder component) was found to be constant and very small, not growing with the load on the controllers. More importantly, the system exhibited near-linear horizontal scalability: the capacity of the controller system, measured by its ability to meet defined Service Level Objectives (SLOs) for queue latency (less than 1 second) and reconciliation latency (less than 5 seconds), increased almost proportionally with each additional controller instance. This perfect horizontal scalability represents a significant breakthrough for high-performance Kubernetes operators.

Technical Deep Dive

▶ Watch: Demonstrating leader election's active-passive HA setup (5:50)

The architecture for horizontally scalable Kubernetes controllers introduces a new core component and leverages existing Kubernetes primitives in innovative ways. At the heart of the system are the controller instances, now referred to as shards, which collectively form a controller ring.

1. The Sharder Component: A new, cluster-wide component called the Sharder is deployed. Its primary responsibilities include:

• Discovering Shards: The Sharder watches for Shard Lease resources, which are created by each individual controller instance to declare its active membership in a specific controller ring. These leases carry labels (e.g., controller-ring: web-hosting-operator) that link them to their respective controller configurations.
• Consistent Hash Ring: The Sharder constructs and maintains a consistent hash ring, similar to those used in distributed systems like Cassandra. This ring maps API objects to available shards, ensuring an even distribution and minimizing reassignments during membership changes.
• Mutating Webhook Configuration: For each configured Controller Ring, the Sharder dynamically creates and manages a Mutating Webhook Configuration. This webhook intercepts API object creation and update requests.
• Label Injection: When a new object (e.g., a Website custom resource) is created, the mutating webhook calls the Sharder. The Sharder determines the appropriate shard based on its consistent hash ring and injects a shard label (e.g., shard: <shard-identity>) onto the object. This label explicitly assigns the object to a specific controller instance.
• Owned Resources: A crucial detail is that not only the primary custom resource (e.g., Website) but also all its controlled resources (e.g., Deployments, ConfigMaps, Services that have an ownerReference back to the Website) are also assigned the same shard label. This ensures that a single controller instance is responsible for the entire lifecycle of a logical unit of work, from the top-level resource down to its managed Kubernetes objects.

1. The Controller Ring CRD: A new Custom Resource Definition (CRD) named ControllerRing is introduced. This resource acts as the configuration for the Sharder, declaring:

• Which controller it pertains to (e.g., web-hosting-operator).
• Which API objects (e.g., Website resources) belong to this controller and should be distributed.
• Which controlled resources (e.g., Deployment, Service, ConfigMap) are owned by these primary objects and should also be sharded together.

1. Controller Modifications: To participate in this sharding scheme, existing controllers require minimal but specific modifications:

• Shard Lease Creation: Instead of a traditional single leader election lease, each controller instance must create its own unique Shard Lease. This lease includes a label referencing the ControllerRing and has a unique holder identity for that specific instance. For controller-runtime users, this is simplified by passing a custom shard-lease implementation to the manager's leader election mechanism.
• Watch Cache Filtering: Each controller instance must configure its watch cache (or informer) to only process objects that carry its specific shard label. This is achieved by adding a label selector to the cache options (e.g., manager.Options.Cache.SelectorsByObject). This ensures that each shard only "sees" and reconciles the objects it is responsible for, significantly reducing its memory footprint and processing load.
• Handover Mechanism ("Train" Label): When objects need to be moved between shards—during rebalancing (e.g., adding a new instance) or failover (e.g., an instance going down)—a carefully designed handover mechanism is employed to prevent conflicting reconciliations.
• The Sharder first adds a temporary "train" label (e.g., shard-train: <new-shard-identity>) to the object slated for migration.
• The currently responsible controller instance observes this "train" label. It must then stop reconciling that specific object and acknowledge the handover by removing both the train label and its original shard label from the object.
• Once the labels are removed, the Sharder can safely inject the new shard label, assigning the object to its new instance. This multi-step process ensures that the old instance definitively relinquishes responsibility before the new instance takes over, avoiding any overlap.
• For controller-runtime users, the library provides wrappers for the predicate (to react to the train label) and the reconciler (to remove the labels), abstracting away the handover logic from the controller's business logic.

These architectural components and modifications, particularly the use of Kubernetes labels for object assignment and the explicit handover mechanism, form a robust system for achieving horizontal scalability while maintaining the critical property of single-instance reconciliation per object. The speaker emphasizes that for controller-runtime users, these changes can be implemented in "50 lines or less" through the provided library.

Demo / Proof of Concept

▶ Watch: Proposed design: Sharding mechanisms for horizontal scalability (8:20)

Tim Ebert's presentation included a compelling live demonstration using a web hosting operator to illustrate both the problem of traditional leader election and the efficacy of the sharding solution.

Initially, the demo showed the web hosting operator running as a single instance. It successfully created a Website object for "CubeCon," which in turn provisioned an NGINX deployment, ingress, and service, making the website accessible. When the operator deployment was scaled to two instances, the kubectl get lease command clearly showed only one pod as the holderIdentity of the leader election lease. Creating a new website confirmed that only the original, active leader instance reconciled it, while the second instance remained passive. A failover was then demonstrated by deleting the active leader, showing the passive instance quickly taking over the lease and reconciling objects—confirming the active-passive HA, but not horizontal scaling.

The core of the demo then shifted to the sharded setup:

1. Sharder Deployment: The sharder component was shown running in the cluster.
2. Controller Ring Configuration: A ControllerRing CRD was applied, configuring the sharder for the web-hosting-operator. This configuration specified that Website objects were to be sharded, along with their controlled resources like Deployment, ConfigMap, and Service.
3. Shard Leases: The kubectl get lease command now showed three distinct ShardLease objects, one for each of the three operator instances, each labeled with controller-ring: web-hosting-operator and marked as ready by the Sharder.
4. Label Injection: A Website object was then created. Inspecting its YAML (kubectl get website cubecon-new -o yaml) revealed that the mutating webhook had successfully injected a shard label (e.g., shard: web-hosting-operator-0) onto the object, assigning it to a specific controller instance. Crucially, the speaker also confirmed that the owned Deployment, Ingress, and Service objects for this website also carried the same shard label.
5. Object Distribution: To demonstrate distribution, a script created 50 random Website objects. A kubectl get websites -o custom-columns=NAME:.metadata.name,SHARD:.metadata.labels.shard command revealed that these 50 websites were "roughly equally distributed" across the three active shards, confirming the load-balancing aspect.
6. Failover and Reassignment: The speaker then scaled the operator deployment down from three to two instances. Watching the leases and websites objects, it was evident that the ShardLease for the removed instance was released, and the Sharder marked it as unavailable. All websites previously assigned to that instance were then updated with new shard labels, reassigning them to one of the two remaining active instances.
7. Adding an Instance and Handover: Finally, an instance was added back, scaling the operator deployment from two to three. This demonstrated the handover mechanism. Websites that needed to be moved to the newly added instance first received the temporary "train" label (e.g., shard-train: web-hosting-operator-2). The existing controller instance responsible for that object observed the train label, stopped reconciling, and removed both the train and its original shard label. Only then did the Sharder assign the new shard label to the object, completing the safe migration to the new instance.

The demo effectively illustrated the dynamic nature of the sharding system, its ability to distribute load, handle instance failures, and rebalance objects while ensuring data consistency and preventing conflicting reconciliations.

Defensive Implications

▶ Watch: Understanding the sharded Kubernetes controller architecture (10:00)

The horizontal scaling solution presented by Tim Ebert offers significant defensive implications for anyone operating Kubernetes clusters, particularly at scale, or developing high-volume custom controllers.

Firstly, it directly addresses a fundamental performance bottleneck in Kubernetes controller design. By enabling true horizontal scaling, operators can now provision critical controllers with sufficient capacity to handle a vast number of API objects (e.g., 9,000 objects in the load tests) and high rates of change (e.g., 300 changes per second). This prevents controllers from becoming a single point of contention or failure under heavy load, improving the overall resilience and stability of the Kubernetes control plane. For platforms like STACKIT's Gardener, which manage thousands of clusters, this translates directly into more robust and performant infrastructure for their customers.

Secondly, the solution enhances High Availability (HA) beyond the traditional active-passive model. Instead of having pre-warmed, idle instances, all deployed instances are actively contributing to the reconciliation workload. In the event of an instance failure, the system automatically reassigns its objects to other active instances, ensuring continuous operation without waiting for a passive replica to take over. This "active-active" HA model provides a more robust and responsive system.

Thirdly, the use of standard Kubernetes API machinery (labels, webhooks, leases) for the sharding mechanism means that the solution integrates natively with existing Kubernetes tooling and workflows. This reduces the operational overhead and learning curve compared to solutions requiring external, non-Kubernetes-native distributed coordination services.

However, it's crucial to acknowledge the speaker's own caution: while the project is "ready for usage" and he encourages trying it out, he "wouldn't recommend running in production" without gathering more real-world experience and building a community around it. This is a critical defensive implication: organizations should approach adoption with a mindset of "early adopter with caution." While the load tests show promising results, production environments often expose edge cases and complexities not fully captured in controlled experiments.

Defenders should consider:

• Testing Rigorously: Implement thorough testing in staging environments, mimicking production loads and failure scenarios to validate the solution's stability and performance.
• Monitoring: Ensure comprehensive monitoring of the Sharder component itself, the ControllerRing resources, ShardLeases, and controller-specific metrics (queue latency, reconciliation latency) to quickly identify and troubleshoot issues.
• Gradual Rollout: For existing operators, consider a gradual rollout strategy, perhaps starting with less critical workloads or a subset of objects, before full-scale deployment.
• Contribution and Feedback: Engaging with the project by providing feedback, bug reports, and contributing to its development will help mature the solution for broader production readiness.

In essence, this work provides a powerful tool for scaling Kubernetes controllers, but its strategic adoption requires a balance of enthusiasm for innovation and prudent operational readiness.

Key Takeaways

• Traditional Kubernetes controllers are fundamentally limited in horizontal scalability due to the reliance on leader election, which ensures only one instance is active, leading to active-passive HA rather than true performance scale-out.
• Sharding, inspired by distributed database principles, provides a robust solution for horizontally scaling Kubernetes controllers, allowing multiple instances to actively reconcile different subsets of API objects concurrently.
• The proposed design leverages native Kubernetes API machinery, utilizing a dedicated Sharder component, ControllerRing CRD, ShardLeases, mutating webhooks for label injection, and label selectors for watch cache filtering.
• A critical handover mechanism using a temporary "train" label ensures conflict-free object migration during rebalancing or failover, preventing multiple controller instances from reconciling the same object simultaneously.
• Implementation is simplified for controller-runtime users through a reusable library, requiring minimal code changes (estimated at 50 lines or less) to integrate sharding capabilities into existing or new operators.
• Load tests demonstrate near-linear scalability with minimal overhead, significantly increasing the system's capacity to handle high object counts (up to 9,000 objects) and churn rates (up to 300 changes per second) while maintaining low latency.
• This project offers a path to build high-performance, resilient Kubernetes controllers, but the speaker encourages community adoption and further real-world testing before widespread production use, emphasizing the need for feedback and experience gathering.

About the Speaker(s)

Tim Ebert is a key member of the STACKIT Kubernetes Engine team at STACKIT. His professional focus revolves around operating Kubernetes at an immense scale, specifically managing thousands of Kubernetes clusters for customers. This challenging environment, built upon the open-source Gardener project, directly inspired his deep dive into the scalability of Kubernetes controllers. His insights and the solution presented in this talk are a direct result of his practical experience and his master's thesis research, "Horizontally Scalable Kubernetes Controllers." He is passionate about improving the fundamental building blocks of Kubernetes for large-scale deployments.