Chaos Engineering Practice Under Ultra-large-scale Cloud Native Edge Computing - Yue Bao & yue li

Chaos Engineering Practice Under Ultra-large-scale Cloud Native Edge Computing

Speakers: Yue Bao, Maintainer of Kubedge Community, Huawei; yue li, Software Quality Engineer, Chair of SIG Testing of Kubedge Community

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=CK7Il4ZiqTA

Overview

This talk, presented by Yue Bao on behalf of yue li, delves into the critical challenges and solutions for achieving stability and scalability in ultra-large-scale cloud-native edge computing environments using Kubedge. As industries rapidly adopt 5G, Industrial IoT, and AI, the number of smart devices at the edge is exploding, creating unprecedented demands for unified management and operation. Kubedge, as the industry's first cloud-native edge computing framework, provides a consistent experience for cloud-edge collaboration across resources, data, devices, and intelligence.

The presentation focuses on the rigorous testing methodologies and architectural optimizations that enable Kubedge to manage an astonishing 100,000 edge nodes and 1 million pods from a single cluster. This scale is not merely theoretical; it addresses real-world demands from projects like high-speed toll stations and vehicle cloud collaborative platforms, where millions of edge nodes (cars) are anticipated. The talk highlights how Kubedge not only meets but exceeds the stringent Service Level Indicators (SLIs) and Service Level Objectives (SLOs) defined by the Kubernetes community, ensuring robust performance even in challenging edge network conditions.

The importance of this work cannot be overstated. With Gartner predicting that smart devices at the edge will outnumber traditional IT devices by 20 times in 2023, and the continued integration of advanced AI capabilities into edge devices by 2028, scalable and stable edge infrastructure is foundational for the digital transformation across smart cities, intelligent manufacturing, and smart healthcare. Kubedge’s recent graduation to a CNCF project further underscores its maturity and significance in the cloud-native ecosystem, providing a proven framework for organizations navigating the complexities of large-scale edge deployments.

Background

▶ Watch: Introduction and challenges of ultra-large-scale edge computing (0:00)

The rapid advancement of 5G networks, the Industrial Internet of Things (IoT), and Artificial Intelligence (AI) has propelled edge computing to the forefront of digital transformation strategies. Future scenarios such as smart cities, smart transportation, smart healthcare, and intelligent manufacturing are becoming increasingly reliant on processing data closer to its source, leading to a significant surge in interest and investment in edge technologies. Gartner's projections underscore this trend, noting that in 2023, the number of smart devices at the edge surpassed traditional IT devices by more than 20 times. By 2028, this growth will continue, with edge devices steadily integrating advanced capabilities like sensors, storage, computing, and AI.

This proliferation of IoT devices presents formidable challenges, primarily due to their diverse types and sheer numbers. Managing and maintaining these vast, distributed fleets uniformly becomes a complex undertaking. The Kubedge community, recognizing these challenges, has faced escalating demands from users requiring solutions for N-scale edge computing, specifically for managing thousands to millions of edge nodes and their associated applications. Concrete examples illustrate this need: a high-speed toll station project based on Kubedge connects nearly 100,000 edge nodes and over 500,000 edge applications across the country. More ambitiously, a vehicle cloud collaborative management platform, treating each car as an edge node, is expected to scale to millions of edge nodes.

Against this backdrop, Kubedge emerges as a crucial solution. It is recognized as the industry's first cloud-native edge computing framework, specifically designed for cloud-edge collaboration scenarios. The project's core mission is to provide a consistent experience for resource, data, device, and intelligent collaboration between the cloud and the edge. Its success is evident in its community growth, boasting over 8,000 GitHub stars, 2,200 forks, and more than 1,500 contributors from over 100 organizations worldwide. A significant milestone was achieved in October of the previous year when Kubedge became a CNCF graduated project, signifying its maturity, stability, and widespread adoption.

The architecture of Kubedge seamlessly connects the cloud, edge, and devices. In the cloud, Kubedge is built directly upon native Kubernetes, making no modifications to the core Kubernetes components. This allows users to leverage standard Kubernetes APIs for managing edge resources. The cloud component, known as Cloud Core, plays a pivotal role. It continuously list-watches the Kubernetes cluster for changes, particularly those related to edge resources, and then transmits this metadata to the edge. On the edge side, the corresponding component is Edge Core. The communication between Cloud Core and Edge Core is facilitated by a robust cloud-edge messaging channel, designed for stability and efficiency. Edge Core itself integrates a lightweight Kubelet, which retains the essential functionalities of Kubernetes while stripping away features not typically required at the edge. This design ensures that native Kubernetes interfaces are fully supported at the edge, enabling the management of edge containers through the Container Runtime Interface (CRI). Furthermore, the architecture supports a wide array of devices, allowing those based on various protocols to connect directly to Edge Core and communicate with the broader cluster.

Key Findings

▶ Watch: Understanding Kubedge's cloud-edge-device architecture (5:00)

The comprehensive scalability testing performed on Kubedge yielded significant findings, demonstrating its robust capability to operate stably at an unprecedented scale. The primary discovery is that a single Kubedge cluster can stably manage 100,000 edge nodes and over 1 million pods simultaneously. This achievement represents a substantial leap in the practical application of cloud-native principles to vast, distributed edge environments.

Crucially, this scalability was validated against stringent Service Level Indicators (SLIs) and Service Level Objectives (SLOs), many of which are reused or adapted from the Kubernetes community's own definitions. The test results confirmed that Kubedge successfully met all defined SLOs:

• API Call Latency: For modification APIs, the latency was consistently less than 100 microseconds. For read-only APIs, the latency was less than 1 second. These figures align with and satisfy the Kubernetes community's requirements for API concurrency. For long streaming read-only APIs, the P99 latency was within the 1-30 second range, depending on the scope.
• Pod Standup Latency: This is a critical metric for edge environments where rapid application deployment is often necessary. The end-to-end P99 latency for pod standup was measured at 4,087 microseconds (approximately 4.087 seconds). This comfortably meets the defined SLO requirement of less than 5,000 microseconds (5 seconds).

These results collectively demonstrate that Kubedge not only supports a massive number of edge nodes and pods but also maintains high performance and responsiveness, essential for critical edge workloads. The successful adherence to these SLIs and SLOs provides strong evidence that Kubedge offers a reliable and production-ready foundation for even the most demanding large-scale edge deployments.

Beyond the immediate test results, the findings also highlighted Kubedge's potential for even greater scaling. The talk mentioned that by leveraging the inherent online/offline ratio of edge nodes (which are not always connected to the cloud) and by employing data sharding technology to distribute resources across corresponding etcd storage, even larger scales than 100,000 nodes and 1 million pods could be achieved. This suggests a flexible and extensible architecture capable of adapting to future growth and diverse operational models.

Technical Deep Dive

▶ Watch: Defining Service Level Indicators and Objectives for scalability (7:00)

Achieving stable operation at a scale of 100,000 edge nodes and 1 million pods requires significant architectural innovation beyond native Kubernetes. The core challenge in traditional Kubernetes clusters at such scales lies in the list-watch mechanism. Every node must list-watch its assigned pods, services, and other metadata, leading to a rapid increase in requests to the Kube API server as the number of nodes and pods grows, ultimately overwhelming it. Kubedge addresses this through several key technical optimizations:

1. Efficient Cloud-Edge Message Channel

Kubedge replaces the direct list-watch paradigm at the edge with a highly efficient, custom-built communication channel. This bidirectional, multipath cloud-edge channel supports both WebSockets and QUIC protocols, providing flexibility and robustness.

• Edge Core Initiated Connections: Unlike traditional nodes, Edge Core initiates connections to Cloud Core.
• Cloud Core Proactive Pushing: Cloud Core performs the list-watch operations on Kubernetes resources in the cloud. Instead of edge nodes pulling data, Cloud Core proactively pushes relevant metadata changes to the respective Edge Cores through this dedicated channel.
• Reduced API Server Load: This design significantly reduces the direct list-watch load on the Kube API server, as edge nodes no longer directly query it, thereby improving overall cluster performance and stability at scale.

2. Reliable and Incremental Cloud-Edge Data Transmission

Edge network environments are inherently challenging, characterized by complex topologies, high latency, intermittent disconnections, and frequent reconnections. In such scenarios, if an edge node reconnects and immediately requests all its data from the cloud, it can create a "forest request" problem, again overwhelming the Kube API server. Kubedge tackles this with an incremental data push model:

• Metadata Version Tracking: Cloud Core maintains a record of the latest metadata version successfully sent to each specific edge node.
• Incremental Updates on Reconnection: When an edge node reconnects, Cloud Core intelligently sends only the incremental data – the changes that have occurred since the last successful transmission.
• Enhanced Stability: This approach prevents an avalanche of requests upon reconnection, significantly improving stability and performance in high-latency, low-quality networks.

3. Lightweight Edge Core

To ensure efficient operation on resource-constrained edge devices, Kubedge developed a lightweight Edge Core. This component is a trimmed-down version of the native Kubelet, optimized for edge environments:

• Feature Removal: Edge Core removes unused features from the native Kubelet, such as in-tree volumes and cloud providers, which are typically irrelevant at the edge.
• Status Information Compression: It compresses status information before transmission, further reducing communication pressure.
• Optimized Resource Usage: Through these optimizations, Edge Core requires a minimum of only 70 megabytes of memory. This minimal footprint allows Kubedge to run stably on a wide range of edge hardware, even in environments with high latency and jitter.

Service Level Indicators (SLIs) and Service Level Objectives (SLOs)

For the scalability testing, Kubedge defined clear SLIs and SLOs:

• Kubernetes-defined SLOs: Reused for API call concurrency. For instance, P99 latency for most API calls should be less than 1 second, and for long-streaming read-only APIs, P99 latency should be between 1 and 30 seconds, depending on the scope.
• Kubedge-specific SLOs: A critical addition is the pod standup latency, requiring the latency for a pod to reach a ready status to be less than 5 seconds (P99).

These precise metrics ensure that "scalability" is not merely about the number of nodes but also about maintaining expected service quality.

Scalability Dimensions

The talk emphasized that scalability is multi-dimensional and not solely based on the number of nodes. While Kubernetes officially supports clusters up to 5,000 nodes and 150,000 pods, Kubedge faces different, more extreme scenarios. For this specific test, based on classic application use cases, Kubedge targeted and validated the ability to manage 100,000 edge nodes and 1 million pods while meeting all defined SLIs and SLOs. This focused approach allowed them to address the most pressing scalability needs for their user base.

Demo / Proof of Concept

▶ Watch: Using HMAC to simulate large-scale Kubedge clusters (10:30)

The talk detailed a rigorous testing methodology and the tools employed to validate Kubedge's scalability and stability, effectively serving as a proof of concept for its architecture.

Testing Deployment Model

The test environment was meticulously configured to simulate a large-scale edge deployment:

• Kubernetes Management Plane: A single-master deployment was used for the core Kubernetes components. This included etcd, kube-API server, kube-scheduler, and kube-controller-manager, each running as a single instance. This configuration represents a typical Kubernetes control plane setup.
• Kubedge Management Plane: To handle the immense scale of edge connections, the Kubedge management plane deployed five instances of Cloud Core.
• Virtual Edge Node Connection: Virtual edge nodes (simulated) were configured to connect randomly to one of these five Cloud Core instances via a node balancer. This setup mimicked a real-world scenario where edge devices would distribute their connections across available cloud components.
• Version Specifics: The testing leveraged specific versions of Kubernetes and Kubedge, with details on QPS (Queries Per Second) and burst parameters for components like kube-scheduler, kube-controller-manager, and various Cloud Core worker configurations (e.g., update port status workers). Further in-depth configuration details were referenced to the official Kubedge scalability testing report.

Testing Tools

Two primary open-source tools were instrumental in conducting and verifying the scalability tests:

1. HMAK (EdgeMark):

• Purpose: HMAK is analogous to Kubernetes' KubeMark, but specifically adapted for Kubedge clusters. It is designed for large-scale scalability testing.
• Mechanism: HMAK operates by deploying special "HMAC pods" within a virtual cluster (referred to as the "HMAC Kubernetes master"). When these pods start, they register themselves as virtual edge nodes in the actual Kubernetes master cluster where Kubedge is deployed.
• Benefit: This approach allows for the simulation of a massive number of edge nodes and pods using limited physical resources. By doing so, HMAK effectively exposes cluster management issues that only manifest in truly large-scale (N-scale) scenarios, enabling pre-emptive identification and resolution.

1. Cluster Node Tool:

• Purpose: This is an open-source Kubernetes node testing tool used to verify whether a cluster meets various service quality standards, particularly against defined SLIs and SLOs.
• Functionality: It provides synchronized data crucial for diagnosing cluster issues and optimizing performance. The tool generates comprehensive Kubernetes cluster performance reports, detailing results across various performance metrics. This was vital for objectively measuring API call latency and pod standup latency.

Test Results

The results of this extensive testing unequivocally demonstrated Kubedge's capabilities:

• API Latency: As previously highlighted, API latency metrics consistently met the defined SLOs. Modification APIs showed latencies under 100 microseconds, while read-only APIs were under 1 second.
• Pod Standup Latency: The end-to-end P99 pod standup latency was measured at 4,087 microseconds, comfortably within the 5,000-microsecond SLO.
• Overall Stability: The comprehensive test results confirmed that both API call concurrency and pod standup latency met the stringent requirements set by the Kubernetes community, proving that Kubedge can stably support 100,000 edge nodes online and manage over 1 million pods.

Further Scaling Potential

The talk also touched upon avenues for achieving even greater scale:

• Online/Offline Ratio: In production environments, edge nodes are often not continuously connected to the cloud (e.g., for security partitions, maintenance). By accounting for the online or offline ratio of edge nodes, the effective scale managed by a single cluster can be further expanded.
• Data Sharding: Implementing data sharding technology to store different resources in corresponding etcd storage instances was identified as a powerful method to achieve even larger scales beyond the validated 100,000 nodes and 1 million pods.

Defensive Implications

▶ Watch: Cluster Node Tool for verifying SLIs and diagnosing issues (11:30)

The detailed insights into Kubedge's scalability and stability offer crucial defensive implications for organizations deploying and managing cloud-native edge environments. Understanding these technical underpinnings allows defenders to build more resilient, performant, and secure edge infrastructures.

For Kubedge Adopters and Operators:

1. Leverage Incremental Updates for Network Resilience: Kubedge's incremental data push model is a significant defensive advantage against network instability. Defenders should design their edge applications and network monitoring to account for and benefit from this behavior. This reduces the blast radius of intermittent connectivity issues and prevents cascading failures on the cloud API server during mass reconnections.
2. Optimize Edge Core Deployment: The lightweight Edge Core (requiring only 70MB of memory) means that edge devices can be more resource-constrained without compromising stability. This allows for a wider selection of cost-effective or specialized hardware, but also implies that defenders must ensure that this minimal footprint is maintained by avoiding unnecessary agents or processes that could bloat the edge runtime.
3. Monitor Cloud-Edge Messaging Channel: The bidirectional, multipath cloud-edge channel (WebSockets and QUIC) is the lifeline of Kubedge. Defenders must implement robust monitoring for the health, latency, and security of these connections. Anomalies in this channel could indicate network issues, misconfigurations, or even potential attack vectors targeting the communication fabric.
4. Design for Disconnected Operations: Kubedge is inherently designed to handle intermittent connectivity. Defenders should embrace this by ensuring edge applications are built with offline capabilities or local caching mechanisms to maintain functionality when disconnected from the cloud. This reduces reliance on constant cloud connectivity and enhances overall system resilience.
5. Proactive Scalability Testing: The use of tools like HMAK (EdgeMark) and Cluster Node Tool is not just for development; it's a critical defensive practice. Defenders should integrate these or similar tools into their CI/CD pipelines and operational readiness testing to validate their specific deployments against defined SLIs/SLOs before and after major changes. This helps identify bottlenecks and vulnerabilities at scale before they impact production.
6. Consider Data Sharding for Extreme Scales: For deployments aiming to exceed 100,000 nodes or 1 million pods, defenders should proactively plan for data sharding technology. This architectural decision, while complex, is essential for maintaining performance and stability and preventing a single point of failure in the etcd backend.

For Cloud-Native Edge Architects and Security Teams:

1. Prioritize Edge-Specific SLOs: The talk emphasized defining SLIs and SLOs tailored for edge scenarios (e.g., pod standup latency). Security teams should collaborate with architects to ensure these SLOs also encompass security-related metrics, such as latency for policy enforcement, certificate revocation checks, or security agent updates, especially in high-latency environments.
2. Robustness as a Security Feature: The stability and resilience provided by Kubedge's architectural optimizations (efficient messaging, incremental updates, lightweight components) directly contribute to a more secure system. A stable platform is less prone to operational errors, misconfigurations, and resource exhaustion, which can often expose vulnerabilities or create opportunities for denial-of-service attacks.
3. Secure the Communication Channel: While not explicitly a security talk, the robust WebSockets and QUIC protocols used in the cloud-edge channel inherently offer better security properties (e.g., encryption, multiplexing, reduced handshake overhead) than less modern alternatives. Defenders must ensure these protocols are correctly configured with strong TLS/SSL, appropriate authentication, and authorization mechanisms.
4. Edge Node Identity and Access Management: With 100,000+ edge nodes, managing identities and access becomes paramount. The Kubedge architecture enables centralized management, but security teams must ensure that each Edge Core and device connecting to it has a strong, verifiable identity and that access policies are strictly enforced, especially given the potential for intermittent connectivity.

By understanding and applying these defensive implications, organizations can harness the power of Kubedge to build highly scalable, stable, and secure cloud-native edge infrastructures capable of meeting the demands of the future.

Key Takeaways

• Unprecedented Scalability Validated: Kubedge has been rigorously tested and proven to stably manage 100,000 edge nodes and over 1 million pods from a single cluster, addressing the demanding requirements of ultra-large-scale edge computing scenarios.
• Adherence to Strict SLOs: This massive scale is achieved while consistently meeting stringent Service Level Objectives, including API call latency (P99 < 1 second for read-only, < 100 microseconds for modification) and critical pod standup latency (P99 of 4,087 microseconds, well within the 5-second target).
• Three Pillars of Optimization: Kubedge's stability at scale is primarily attributed to three core architectural innovations: an efficient cloud-edge message channel utilizing WebSockets and QUIC, a reliable incremental data push model for resilient data transmission in challenging networks, and a lightweight Edge Core requiring only 70MB of memory.
• Specialized Testing Tools are Crucial: Tools like HMAK (EdgeMark) and the Cluster Node Tool are essential for simulating and validating large-scale edge environments, enabling the identification and resolution of scalability issues that only manifest at extreme scales.
• CNCF Graduated Project for Production Readiness: As a CNCF graduated project, Kubedge offers a mature, robust, and community-backed framework for organizations looking to implement cloud-native edge solutions, providing a consistent experience for cloud-edge collaboration across various dimensions.
• Future Focus on Edge-Specific Challenges: Future development will concentrate on defining new SLIs and SLOs tailored for edge devices, edge-cloud messaging, and edge service meshes, particularly to evaluate service quality in scenarios with network disconnections and high latency.

About the Speaker(s)

Yue Bao is a maintainer of the Kubedge community and works at Huawei. He presented this talk at KubeCon EU, stepping in for his colleague. His expertise lies in ensuring the stability and scalability of cloud-native edge computing solutions.

yue li is a Software Quality Engineer and serves as the Chair of the SIG Testing within the Kubedge community. Her work focuses on the rigorous testing and quality assurance of Kubedge, particularly concerning its performance and scalability under large-scale conditions. Due to scheduling conflicts, yue li was unable to attend KubeCon EU in person, and Yue Bao presented on her behalf.