Multi-cluster Orchestration System: Karmada Updates and Use Cas... Hongcai Ren & Joe Nathan Abellard

Multi-cluster Orchestration System: Karmada Updates and Use Cases

Speakers: Hongcai Ren, Maintainer, Huawei; Joe Nathan Abellard, Cloud Compute Services Platform Team, Bloomberg

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=rbVV8WIJYww

Overview

This session provides a comprehensive update on Karmada, an open-source project designed for multi-cluster application management and orchestration, alongside a detailed exploration of its real-world implementation and significant contributions from Bloomberg. Hongcai Ren, a Karmada maintainer from Huawei, introduced the core capabilities of Karmada, positioning it as a robust solution for deploying and managing workloads across a fleet of Kubernetes clusters. Joe Nathan Abellard from Bloomberg then delved into how his team built a managed service for Karmada to support their diverse and demanding internal use cases, highlighting the practical challenges and innovative solutions developed in an enterprise environment.

The talk underscores the growing complexity of operating Kubernetes at scale, particularly when spanning multiple clusters and data centers. It demonstrates how Karmada addresses critical needs such as workload scheduling, global resource visibility, and advanced failover mechanisms for both applications and entire clusters. Furthermore, Bloomberg's experience illuminates the journey from adopting an upstream project to actively contributing substantial enhancements driven by specific enterprise requirements like air-gapped environments, stringent security policies, and high-availability demands. This session is crucial for anyone involved in multi-cluster Kubernetes deployments, offering insights into both the foundational capabilities of Karmada and the operational strategies required to leverage it effectively in a production setting.

Background

▶ Watch: Talk Introduction and Agenda Overview (0:00)

The proliferation of Kubernetes across various environments—different cloud providers, on-premises data centers, or even edge locations—has led to a fragmented operational landscape. While Kubernetes excels at managing applications within a single cluster, orchestrating workloads, configurations, and policies consistently across dozens or hundreds of clusters introduces significant complexity. This challenge is precisely what Karmada, a project open-sourced in 2021, aims to solve. It acts as a multi-cluster orchestration system, allowing users to manage applications centrally and distribute them across a federation of clusters.

Before Karmada, organizations often resorted to custom scripts, GitOps tools applied per cluster, or rudimentary federation approaches, which typically lacked the comprehensive scheduling, policy enforcement, and failover capabilities inherent in a dedicated multi-cluster orchestrator. Karmada positions itself as a Continuous Delivery (CD) system for multi-cluster environments, enabling administrators to deploy workloads to Karmada, which then intelligently schedules them to one or more member clusters. It also provides a global view of resources, which can be reported to third-party databases like Elasticsearch or OpenSearch for centralized monitoring and auditing.

Bloomberg, as detailed by Joe Nathan Abellard, faces these multi-cluster challenges acutely, driven by a wide array of demanding use cases:

• Reliable Resource Propagation: The fundamental need to consistently propagate Kubernetes resources such as ConfigMaps, Secrets, ClusterWorkflowTemplates, and WorkflowTemplates to a set of clusters.
• AI Inference Acceleration: Federating mo cache resources on GPU nodes across clusters to reduce model warm-up times, critical for accelerating AI inference workloads.
• Stateful Application Failover: Improving the resiliency of Apache Flink workloads on multi-cluster Kubernetes by enabling stateful application failover, an advanced capability for distributed streaming applications.
• GPU Utilization Optimization: Optimizing the utilization of GPUs for AI model training across heterogeneous clusters, addressing the challenge of managing diverse hardware environments efficiently.

Given these complex and mission-critical requirements, Bloomberg recognized that simply using upstream Karmada would not suffice. The overhead of each application team building and maintaining their own Karmada multi-cluster infrastructure was prohibitive. This led to the development of a managed service offering for Karmada, abstracting away the underlying complexities and providing a simplified, enterprise-grade solution for internal tenants.

Key Findings

▶ Watch: Karmada's Cluster and Application Failover Feature (2:00)

The presentation delivered several key findings regarding the capabilities of Karmada and its practical application in an enterprise setting:

1. Karmada as a Comprehensive Multi-cluster Orchestrator: Karmada effectively serves as a central control plane for managing applications across a distributed fleet of Kubernetes clusters. Its core features, including intelligent scheduling, resource propagation, global resource visibility, and both cluster-level and application-level failover, are critical for achieving high availability and operational consistency in multi-cluster environments. The project's evolution to incubation status within the CNCF and its growing community of over 700 contributors and 36 public adopters underscore its maturity and widespread adoption.

1. Necessity of a Managed Service Layer for Enterprise Adoption: For large organizations like Bloomberg, the raw complexity of deploying and operating Karmada across multiple data centers, integrating it with internal systems, and ensuring high availability necessitates a dedicated managed service. This managed service significantly simplifies the consumption of multi-cluster capabilities for internal teams, transforming a complex infrastructure into a trivial "two-step" onboarding process for tenants. This approach offloads the burden of infrastructure management from application developers, allowing them to focus on their core business logic.

1. The "Stretch Topology" for High Availability: Bloomberg's innovative "stretch topology" provides a robust solution for ensuring the high availability of the Karmada control plane itself. By distributing the control plane across multiple host clusters spanning different data centers and backing it with a multi-data center Etcd cluster, this architecture can withstand various failure scenarios, including individual member cluster failures, host cluster failures, and even the loss of an entire data center, while maintaining continuity of service.

1. Real-world Requirements Drive Open-Source Contributions: The specific operational and security requirements of an enterprise like Bloomberg have led to significant contributions back to the upstream Karmada project. These contributions address critical needs such as support for air-gapped environments (custom HTTP sources for CRD manifests), enhanced security (configurable priority classes for components, custom CA certificates, KMS integration via API server sidecar containers, configurable certificate validity periods), and improved integration with external systems (external Etcd, API server service discovery, extra volumes/volume mounts). These contributions not only benefit Bloomberg but also enrich the entire Karmada community.

1. Focus on AI/ML Workloads and Usability: Karmada's roadmap demonstrates a strong commitment to supporting AI training and inference jobs, acknowledging the increasing demand for distributed AI workloads. Furthermore, the ongoing development of a Karmada dashboard highlights a focus on improving user experience and making multi-cluster management more accessible through a graphical user interface.

Technical Deep Dive

▶ Watch: Bloomberg's Multicluster Federation Use Cases (3:03)

Karmada operates as a sophisticated control plane atop a fleet of Kubernetes clusters. At its core, Karmada extends the Kubernetes API to manage resources across multiple clusters. When a user deploys a workload to Karmada, its scheduler intelligently determines which registered member clusters are best suited based on policies, resource availability, and other criteria. The resource propagation mechanism then ensures that the intended state is applied to the selected member clusters. For operational visibility, Karmada supports reporting global resource views to external systems like Elasticsearch or OpenSearch. Critical for enterprise resilience are its failover capabilities, which include both cluster failover (migrating applications from a crashed cluster to another) and application failover (restarting or migrating a crashing job within a cluster or to another cluster).

Bloomberg's journey to a managed Karmada service involved building a layered architecture:

1. Host/Management Cluster: This is the foundational cluster where the core Karmada infrastructure resides. The upstream Karmada operator is installed here, responsible for provisioning a base Karmada control plane.

1. Managed Karmada Operator (Bloomberg's Custom Component): This custom operator is central to Bloomberg's managed service. It integrates with internal Bloomberg systems to:

• Set up ingress traffic to the managed Karmada control planes, providing a unified access point.
• Manage a cluster registration subsystem for fully automated and dynamic member cluster registration. This subsystem includes:
• Cluster Provider: Integrates with Bloomberg's internal cluster inventory API to dynamically sync cluster state.
• Cluster Credentials Provider: Integrates with Bloomberg's internal token exchange service to securely provide Karmada with the necessary credentials to push workloads to registered member clusters.
• Provision and manage scheduler estimator instances for Karmada's estimator add-on. When enabled, the estimator informs the Karmada scheduler on optimal workload federation, and Bloomberg's scheduler estimator provider handles the provisioning and plumbing for each member cluster.

To achieve high availability, Bloomberg developed a stretch topology for its managed Karmada instances. This architecture involves:

• Multiple Host Clusters: Instead of a single host cluster, multiple host/management clusters are deployed, spanning across multiple physical data centers.
• Stretched Karmada Instance: A single Karmada instance is logically "stretched" across this cohort of management clusters.
• Unified Ingress: Integration with Bloomberg's DNS-based load balancer managed service provides one unified ingress endpoint to each managed Karmada instance. This requires ensuring the same CA certificate is used to provision the Karmada instance on all host clusters.
• Multi-cluster Etcd: Integration with Bloomberg's managed Etcd offering provisions a single, highly available multi-cluster Etcd instance that all Karmada instances across the host clusters connect to. This ensures data consistency and quorum even if an entire data center is lost.

This stretch topology offers significant resilience:

• Member Cluster Failure: Service continuity is maintained as Karmada can migrate workloads to other operational member clusters.
• Single Host Cluster Failure: The Karmada control plane remains operational via the instance running on the other host cluster in a different data center.
• Entire Data Center Failure: While some Etcd members and a host cluster might be lost, the Etcd cluster maintains quorum, and the Karmada instance in the surviving data center continues to operate, ensuring service continuity.

Digging deeper into the stretch topology's implementation, Bloomberg created a Central Source of Truth (SOT). This is a minimal control plane comprising a kube API server, a controller manager (with a subset of controllers), and cert manager components, stacked across the cohort of host clusters. This SOT is also backed by a multi-data center Etcd cluster and accessed via a unified DNS-based load balancer.

The operational flow for provisioning a Karmada instance in this setup is:

1. Custom Resource (CR) Application: A custom resource defining the desired Karmada instance is applied to the SOT API server.
2. Source of Truth Operator Action: The Source of Truth Operator (listening to SOT API server events) allocates an Etcd cluster, provisions the necessary API server CA certificates for unified ingress, and client certificates for the Karmada control plane.
3. Sync Operator: The Sync Operator (also listening to SOT API server events) syncs all these generated resources from the SOT API server to the individual host cluster API servers.
4. Managed Karmada Operator Provisioning: With these resources available on each host cluster, the Managed Karmada Operator can then provision the Karmada instance on that specific host cluster, completing the stretched deployment.

Bloomberg's commitment to Karmada also extended to numerous open-source contributions, driven by specific enterprise requirements:

• Custom HTTP Source for CRD Manifests: To enable Karmada deployment in air-gapped environments, Bloomberg contributed support for the Karmada operator to download CRD manifests from a custom HTTP source, rather than relying on GitHub release artifacts.
• Configurable Priority Class: Added support for configuring the priority class of managed Karmada control plane components, ensuring reliability and stability of critical services.
• Custom CA Certificates: Enabled the Karmada operator to use custom CA certificates for provisioning instances, crucial for the unified ingress endpoint in the stretch topology.
• API Server Sidecar Containers for KMS Integration: To support encryption at rest for confidential data, this contribution added support for API server sidecar containers. This allows a KMS plugin to run alongside the kube API server, bridging integration with an internal key management service via gRPC for encryption/decryption requests.
• Configurable Certificate Validity Period: To align with organizational security policies, a contribution was made to make the validity period of leaf certificates in Karmada's PKI configurable.
• API Server Service Discovery: Added support for the Karmada operator to reliably discover the API server service, which is essential for configuring ingress traffic.
• External Etcd Support: Enabled the Karmada operator to use external Etcd clusters, allowing integration with Bloomberg's managed multi-data center Etcd service.
• Extra Volumes and Volume Mounts: Enhanced the Karmada operator's capabilities by adding support for specifying extra volumes and volume mounts for components. This unlocked use cases such as integrating with internal authn webhooks for authentication and configuring additional encryption at rest mechanisms.

These contributions highlight a practical, iterative approach to integrating open-source projects into a large enterprise, where specific needs lead to valuable enhancements for the entire community.

Demo / Proof of Concept

▶ Watch: Building Bloomberg's Managed Karmada Service Architecture (6:07)

The speakers explicitly noted that due to time constraints, a live demonstration was not possible during the session. Joe Nathan Abellard stated, "my favorite part of doing this is live demos but unfortunately we don't have enough time so gonna pass back to Hungai." Therefore, the talk focused on presenting the architectural designs, use cases, and technical contributions rather than showcasing a live proof of concept.

Defensive Implications

▶ Watch: Introducing Stretch Topology for High Availability (8:36)

For organizations leveraging or considering Karmada for multi-cluster orchestration, the insights from this talk provide several critical defensive implications:

1. Prioritize High Availability of the Karmada Control Plane: The "stretch topology" demonstrated by Bloomberg is a blueprint for ensuring that Karmada itself is not a single point of failure. Defenders should adopt similar strategies, deploying Karmada across multiple availability zones or data centers, utilizing shared, highly available external Etcd instances, and implementing unified ingress with DNS-based load balancing to protect against infrastructure outages.

1. Implement Robust Certificate Management: The ability to configure custom CA certificates and certificate validity periods (a direct contribution from Bloomberg) is crucial for aligning with enterprise security policies. Defenders must ensure that Karmada's PKI is managed securely, with appropriate rotation policies and integration into existing certificate management workflows.

1. Integrate with Enterprise Security Services: Leverage Karmada's extensibility to integrate with internal security services. The support for API server sidecar containers for KMS integration and extra volumes/volume mounts for authentication webhooks are prime examples. This allows confidential data to be encrypted at rest using enterprise-grade key management solutions and ensures that all access to the Karmada control plane adheres to organizational authentication and authorization policies.

1. Harden Karmada Control Plane Components: Utilize features like configurable priority classes for Karmada components to ensure that critical orchestration services receive adequate resources and remain stable under load. This prevents resource starvation that could lead to service disruptions.

1. Prepare for Air-Gapped or Restricted Environments: For environments with strict network isolation, the contribution allowing custom HTTP sources for CRD manifests is vital. Defenders should pre-position necessary artifacts within their isolated networks or leverage internal registries to ensure Karmada can be deployed and updated without external network access.

1. Leverage Failover Capabilities for Application Resilience: Actively design applications to leverage Karmada's cluster and application failover features. This involves understanding how Karmada migrates workloads and configuring applications to be resilient to such migrations, ensuring business continuity even during member cluster failures.

1. Establish Global Visibility and Auditing: Utilize Karmada's ability to report global resource views to centralized logging and monitoring systems (like Elasticsearch or OpenSearch). This provides critical insights for security operations, enabling anomaly detection, compliance auditing, and incident response across the entire multi-cluster estate.

1. Contribute to Upstream Security Enhancements: Organizations with specific security needs, particularly large enterprises, should consider contributing back to the Karmada project. Bloomberg's example demonstrates that addressing internal requirements can significantly enhance the security posture and operational capabilities of the entire community.

Key Takeaways

• Karmada is a mature and growing solution for multi-cluster Kubernetes orchestration, providing essential capabilities like intelligent scheduling, resource propagation, global visibility, and robust failover mechanisms for both clusters and applications.
• Enterprise adoption often requires a managed service layer built atop Karmada to abstract complexity, integrate with internal systems, and streamline operations for application teams, as exemplified by Bloomberg's implementation.
• High availability of the Karmada control plane is paramount, achievable through architectures like Bloomberg's "stretch topology" that distribute Karmada instances across multiple data centers and leverage highly available external Etcd.
• Real-world enterprise requirements drive significant open-source contributions to Karmada, enhancing its capabilities for air-gapped environments, advanced security integrations (KMS, custom CAs, configurable cert validity), and better operational control (external Etcd, API server service discovery).
• Karmada is actively evolving to support demanding AI/ML workloads, focusing on performance, scalability, and improved user experience through initiatives like a dedicated UI dashboard.
• Karmada's multi-cluster-first design differentiates it from single-cluster tools that may offer multi-cluster support as an add-on, making it a purpose-built solution for complex federated Kubernetes environments.

About the Speaker(s)

Hongcai Ren is a maintainer of the Karmada project, working at Huawei. He provided an overview of Karmada's core features and its community growth, highlighting its evolution within the CNCF and its strategic roadmap, particularly focusing on AI/ML workloads, performance, and scalability.

Joe Nathan Abellard is a member of the Cloud Compute Services Platform team at Bloomberg and is also a maintainer of the Karmada operator. He shared detailed insights into Bloomberg's real-world adoption of Karmada, including their innovative managed service architecture, the "stretch topology" for high availability, and numerous significant open-source contributions made to the Karmada project based on Bloomberg's enterprise requirements.