Dynamic Multi-Cluster Controllers With Controller-runtime - Marvin Beckers & Stefan Schimanski

Dynamic Multi-Cluster Controllers With Controller-runtime

Speakers: Marvin Beckers, Team Lead at Kubernetic; Stefan Schimanski, API Server Topics at Nvidia

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Tz8IcMSY7jw

Overview

This talk, presented by Marvin Beckers and Stefan Schimanski, introduces multicluster-runtime, a novel project designed to extend the popular controller-runtime framework with native multi-cluster capabilities. In an era where running multiple Kubernetes clusters is the norm rather than the exception, the challenge of consistently and efficiently reconciling resources across an entire fleet of clusters has become a significant hurdle for platform engineers and developers. While existing solutions offer bespoke approaches, they often lack generality, widespread adoption, and deep integration with the core controller-runtime ecosystem.

Beckers and Schimanski address this critical gap by presenting multicluster-runtime as a friendly, non-forking extension that leverages controller-runtime's primitives and Go generic support. The project, hosted under the Kubernetes SIG Multicluster, aims to standardize multi-cluster controller development by introducing a provider concept for dynamic cluster discovery and management. This approach allows developers to write controllers that can seamlessly operate across diverse cluster topologies, from uniform reconciliation across many identical clusters to more complex non-uniform scenarios involving host and child clusters or specialized central services.

The significance of multicluster-runtime lies in its potential to streamline the development and operation of multi-cluster applications and management planes. By providing a generic, officially backed framework, it seeks to eliminate the need for custom, often fragmented, multi-cluster solutions, thereby reducing development overhead, improving consistency, and fostering broader innovation in the Kubernetes ecosystem. The speakers emphasize that this is an experimental project, actively seeking community feedback to shape its future and ensure it meets the diverse needs of multi-cluster practitioners.

Background

▶ Watch: Why reconcile across multiple Kubernetes clusters? (0:40)

The modern Kubernetes landscape is inherently multi-cluster. Organizations routinely operate dozens, if not hundreds, of clusters for various reasons – isolation, regional deployments, tenancy, or simply scaling beyond the limits of a single cluster. Tools like Cluster API facilitate the lifecycle management of these clusters, while solutions like vcluster offer lightweight, virtual Kubernetes API servers, further solidifying multi-cluster as the de facto standard.

However, the operational challenge lies in reconciling applications and infrastructure across this distributed fleet. While individual clusters are well-served by controller-runtime, a foundational library for building Kubernetes controllers, its design is inherently single-cluster focused. This fundamental mismatch has led to a proliferation of custom solutions for multi-cluster reconciliation. For instance, Cluster API developed its own cluster-cache package. The KCP project, which aims to provide a control plane for multi-cluster and multi-tenant environments, created a fork of controller-runtime to embed multi-cluster capabilities. Older projects like Admirality's multicluster-controller also exist. The common thread among these is their bespoke nature; they are not generic, not widely adopted as a standard, and not natively integrated with controller-runtime in a universal way.

The speakers highlight several existing scaling models for controllers in a multi-cluster environment:

1. Per Kubernetes Cluster: The most traditional approach, where a dedicated controller pod, manager, cache, work queue, and reconciler run independently within each target cluster. This leads to significant resource duplication and management overhead.
2. Management Cluster with One Controller Pod per External Cluster: A central management cluster hosts a process manager (perhaps another operator) that dynamically launches individual controller pods, each responsible for reconciling a single external cluster. This centralizes management but still involves multiple independent controller instances.
3. Host Cluster with One Controller Pod Running Multiple Managers: A single controller pod within a host/management cluster starts multiple controller-runtime managers, each configured to reconcile against a specific external Kubernetes cluster. This brings some centralization but can become complex to manage dynamically.

The motivation behind multicluster-runtime is to move beyond these existing, often fragmented, models by offering a more generic and tightly integrated solution that extends controller-runtime itself. The goal is to provide a standardized, native way for controllers to discover and interact with an arbitrary number of clusters, making multi-cluster reconciliation a first-class citizen in the controller development paradigm.

Key Findings

▶ Watch: Announcing Multicluster Runtime: a controller-runtime extension (4:00)

The core contribution of Beckers and Schimanski's talk is the introduction of multicluster-runtime, a new open-source project designed to be a "friendly extension" of controller-runtime. Crucially, it is not a fork, but rather builds upon the existing primitives and leverages Go's generic support within controller-runtime to add multi-cluster awareness. This project is hosted under the Kubernetes SIG Multicluster on GitHub, signifying its alignment with official Kubernetes community efforts.

The central innovation of multicluster-runtime is the Provider concept. A provider is an interface that abstracts the mechanism by which clusters are discovered and managed. It is responsible for identifying the clusters that a controller needs to interact with, and for providing the necessary client and cache objects for each. This abstraction allows multicluster-runtime to support diverse environments without requiring changes to the core controller logic. Examples of potential providers include those for Cluster API, Kind clusters, cloud providers (e.g., listing all clusters in an AWS account), or even virtual workspaces in projects like KCP. The speakers also note that the "cluster concept" is highly generic, and a provider could even abstract namespaces as "clusters" for isolation or audit logging purposes.

multicluster-runtime supports two main reconciliation topologies:

1. Uniform Reconciler: This is for controllers that are "cluster-agnostic." They perform the same logic for events originating from any cluster, reading and writing resources within the same cluster context. This pattern is ideal for running an existing, single-cluster reconciler against n different clusters simultaneously.
2. Non-Uniform Reconciler: These controllers are explicitly "cluster-aware." They might have a special "host" or "central" cluster and interact differently with "child" or "remote" clusters. Examples include a sinker that synchronizes objects from one cluster to many, or a Cert-Manager instance in a host cluster issuing certificates for multiple child clusters without exposing sensitive credentials to the children.

The architecture introduces a multicluster.Manager which, unlike its single-cluster counterpart, takes a Provider during initialization. This manager dynamically engages and disengages clusters as the provider discovers or removes them, managing the underlying caches and informers for each. The reconciler's Request object is enriched with cluster information, allowing the reconciler function to dynamically fetch the correct cluster client and cache for the specific event it is processing. This design ensures that the core reconciliation logic remains largely similar to single-cluster controllers, requiring only minor adaptations to become multi-cluster aware.

The project is explicitly labeled as an "experiment," and the speakers are actively soliciting feedback from the community. They emphasize that the goal is to build a truly useful and widely adopted standard, avoiding the "another standard" trap, and are keen to understand if the proposed model addresses real-world needs effectively.

Technical Deep Dive

▶ Watch: Multicluster Runtime's core: the Provider concept (4:40)

At the heart of multicluster-runtime is the multicluster.Manager interface, which extends the capabilities of the standard controller-runtime manager. Instead of being tied to a single, statically configured Kubernetes cluster, the multicluster.Manager is initialized with a Provider interface. This provider dynamically informs the manager about the clusters it should be aware of.

Key methods of the multicluster.Manager include:

• GetCluster(clusterName string): This is the primary method for reconcilers to dynamically obtain a controller-runtime cluster.Cluster object (which encapsulates a client and cache) for a specific named cluster. If clusterName is an empty string, it refers to the host cluster where the multi-cluster controller itself is running. This dynamic fetching is crucial because the reconciler no longer operates against a fixed, embedded cluster object but must react to requests originating from various clusters.
• GetManager(clusterName string): For compatibility, this method allows obtaining a standard controller-runtime.Manager scoped to a specific cluster. This can be useful for integrating existing single-cluster logic or when a reconciler needs the full manager interface for a particular cluster.
• GetProvider(): Returns the underlying provider, which can be useful for advanced scenarios like managing indexes across dynamically discovered clusters.

The Provider interface is what developers implement to define how multicluster-runtime discovers and connects to clusters. It's designed to be lightweight and flexible:

• GetCluster(clusterName string): Similar to the manager's method, this is where the provider's logic resides for creating or retrieving a controller-runtime cluster.Cluster object for a given clusterName. Providers are expected to maintain state (e.g., a map of running caches) to avoid recreating connections and caches repeatedly.
• ManageIndexes(indexers cache.Indexer): This method ensures that any index definitions required by the reconciler are applied to new caches as clusters come online. This is critical for maintaining consistent indexing across a dynamic fleet of clusters.

When a provider discovers a new cluster, it "engages" it with the multicluster.Manager. The manager then sets up the necessary sources (e.g., informers, caches) for that cluster. These sources listen for events and feed them into a shared work queue. Conversely, when a cluster goes offline, the provider "disengages" it, typically by cancelling the associated context, which signals the manager to tear down its resources for that cluster.

The adaptation for a multicluster-runtime reconciler is minimal but significant:

1. Builder Package: Instead of controller-runtime/pkg/builder, developers use multicluster-runtime/pkg/builder.
2. Reconcile Request: The Reconcile function receives a request type from multicluster-runtime/pkg/reconcile which embeds the standard controller-runtime request, but crucially adds a ClusterName field.
3. Dynamic Client/Cache Access: Inside the Reconcile function, the reconciler uses r.MultiClusterManager.GetCluster(request.ClusterName) to obtain the specific cluster.Cluster object for the cluster that originated the event. From this cluster object, it can then access the client (cluster.GetClient()) and cache (cluster.GetCache()) for that particular cluster.

A critical consideration highlighted by the speakers is bottleneck management and fair queuing. In a multi-cluster setup, one highly active cluster could potentially flood the shared work queue, starving events from other, less active clusters. To mitigate this, the concept of fair queuing is introduced:

• Instead of a single global queue, the idea is to have individual queues per cluster.
• A fair queuing mechanism would then pick events from these individual queues in a balanced manner, ensuring no single cluster can monopolize the reconciler's capacity.
• This could also involve priority queues or other advanced queuing strategies, similar to those found in controller-runtime.
• The specific queuing strategy would depend on the controller's bottlenecks, which could be client QPS, CPU consumption, or worker capacity. This aspect is still an area for further development and community input.

The fact that multicluster-runtime is built as an extension and not a fork is a testament to controller-runtime's architecture, particularly its support for Go generics. This allows multicluster-runtime to inject its multi-cluster aware types while still leveraging the underlying controller-runtime framework, ensuring compatibility and reducing maintenance overhead.

Demo / Proof of Concept

▶ Watch: Understanding uniform and non-uniform reconciler topologies (5:20)

Marvin Beckers demonstrated the practical application of multicluster-runtime through a live coding session, showcasing its ease of use and flexibility. The demo centered around a simple controller designed to log details of Config Maps it discovers. The beauty of the demonstration was in its ability to switch between different cluster providers with minimal code changes.

Initially, the controller was configured to use a KCP provider. KCP workspaces, while logically distinct, often run on a single KCP instance and are presented as virtual clusters. The controller logs showed lines indicating reconciliation events for Config Maps (specifically kube-root-ca.crt) from various "cluster names," each corresponding to a different KCP workspace. This illustrated the multicluster-runtime's ability to treat these logical separations as distinct clusters and process their resources independently within a single controller instance.

The most compelling part of the demo was the seamless transition to a different provider. Marvin showed that by changing just two lines of code—importing a different provider package and instantiating the Kind provider instead of the KCP one—the controller immediately adapted. After recompiling and restarting, the controller began logging Config Maps from a local Kind cluster running on the machine. The dynamic nature was further emphasized by the ability to start additional Kind clusters, which the controller would then automatically discover and begin reconciling against, without any further code changes or restarts to the controller itself.

To highlight the simplicity of implementing a provider, Marvin also walked through the code for the Kind provider. This provider, remarkably, comprised only about 200 lines of Go code. Its core logic involved:

1. Periodically listing available Kind clusters (e.g., using kind get clusters).
2. For each discovered cluster, generating a kubeconfig.
3. Using this kubeconfig to set up a controller-runtime client and cache.
4. Storing and managing these client/cache objects in a map, as Stefan had previously described, to avoid recreating them on every request.

The speakers clarified that while a few prototype providers (Kind, KCP, Cluster API, experimental namespace provider) exist in the multicluster-runtime repository, the long-term goal is not to create a monorepo for all providers. Instead, more complex and domain-specific providers, such as a robust Cluster API provider, would ideally reside in their respective projects or dedicated repositories, leveraging the multicluster-runtime interfaces. This approach promotes modularity and allows for specialized development within different ecosystems.

The demo effectively showcased the project's core promise: abstracting multi-cluster complexities behind a simple provider interface and enabling existing controller-runtime developers to quickly adapt their logic for a multi-cluster world.

Defensive Implications

▶ Watch: Architecture: Multicluster Manager and Provider interaction (7:20)

While multicluster-runtime is fundamentally a developer tool aimed at improving operational efficiency and consistency, its design has significant indirect defensive implications for Kubernetes environments.

Firstly, by providing a generic and standardized framework for multi-cluster reconciliation, it encourages the adoption of unified management patterns. Instead of disparate, custom scripts or single-cluster controllers for each cluster, organizations can deploy a single, well-tested multicluster-runtime controller. This inherently leads to improved consistency in applying security policies, configurations, and desired states across an entire fleet of clusters. Inconsistent configurations are a common source of security vulnerabilities, and multicluster-runtime helps mitigate this risk.

Secondly, the centralized nature of a multicluster-runtime controller, operating from a host or management cluster, can simplify auditing and control. Instead of needing to monitor many individual controller instances scattered across various clusters, security teams can focus their attention on the single multicluster-runtime controller. This central point can be instrumented for robust logging, monitoring, and access control, providing a clearer picture of actions being performed across the entire cluster fleet. The ability to abstract "clusters" as namespaces or tenants within a provider (as mentioned conceptually in the talk) further enhances isolation and fine-grained auditing capabilities, allowing security teams to enforce policies specific to certain tenants or logical boundaries.

Thirdly, the project's focus on fair queuing for events directly addresses a potential security and availability concern. Without fair queuing, a misbehaving or compromised cluster generating an excessive number of events could effectively launch a denial-of-service (DoS) attack against the multi-cluster controller, preventing it from reconciling critical resources in other clusters. By implementing fair queuing, multicluster-runtime aims to ensure that no single cluster can exhaust the controller's capacity, thus maintaining the availability and responsiveness of the management plane for all clusters. This resilience is a key defensive attribute in distributed systems.

Finally, by reducing the complexity of multi-cluster controller development, multicluster-runtime can empower developers to build more secure applications from the ground up. Simpler codebases with fewer custom multi-cluster constructs are generally easier to review, test, and maintain, leading to fewer accidental security flaws. The project's official backing by Kubernetes SIG Multicluster also suggests that it will benefit from community scrutiny and best practices, further contributing to its security posture as it matures.

Key Takeaways

• multicluster-runtime is a new, official Kubernetes SIG Multicluster project that extends controller-runtime to natively support multi-cluster reconciliation without forking.
• It introduces a Provider interface for dynamic cluster discovery and management, allowing controllers to adapt to changing cluster fleets (e.g., Kind, KCP, Cluster API, or even namespace-based "virtual clusters").
• Minimal code changes are required to adapt existing controller-runtime reconcilers, primarily involving using multicluster-runtime's builder and reconcile packages, and dynamically fetching cluster objects via the manager.
• The project supports both uniform (cluster-agnostic) and non-uniform (cluster-aware) reconciliation patterns, enabling flexible management of diverse multi-cluster topologies.
• It addresses critical scaling and performance challenges in multi-cluster environments, particularly through the concept of fair queuing to prevent single clusters from monopolizing reconciliation resources.
• multicluster-runtime is currently an experimental project actively seeking community feedback and contributions to refine its design, stabilize its features, and ensure it meets real-world multi-cluster operational needs.

About the Speaker(s)

Marvin Beckers is a Team Lead at Kubernetic. His work on multicluster-runtime has been supported by the IPSIS project, an initiative from the European Union.

Stefan Schimanski is a long-time contributor to Kubernetes API server topics, currently working at Nvidia. He has extensive experience in the Kubernetes ecosystem and, along with Marvin, has a background in the KCP project, which heavily influenced the design and necessity of multicluster-runtime.