Cloud Native AI: Harness the Power of Advanced Scheduling for High-P... William Wang & Xuzheng Chang

Cloud Native AI: Harnessing the Power of Advanced Scheduling for High-Performance Machine Learning with Volcano

Speakers: Xuzheng Chang (Kevin Juan), TOC Member & Maintainer; Shu Jun (Zir), Maintainer (representing William Wang)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=yCyezOTVU_Y

Overview

This talk, presented by Xuzheng Chang (Kevin Juan) and Shu Jun (Zir), delves into the latest advancements within the Volcano project, a Kubernetes-native batch scheduler designed to optimize high-performance AI and machine learning (ML) training workloads. As AI, particularly large language models (LMS), experiences rapid growth, the demand for sophisticated, efficient, and scalable infrastructure has intensified. This presentation addresses the evolving landscape of cloud-native AI, highlighting the critical need for advanced scheduling capabilities that can bridge the gap between complex underlying hardware topologies and the simplicity required by data scientists.

The core of the discussion revolves around Volcano's innovative features, with a particular emphasis on the newly introduced HyperNode API and network topology-aware scheduling. These features aim to significantly accelerate distributed AI training and inference by intelligently mapping workloads to the most performant network configurations. Beyond topology awareness, the speakers also covered crucial aspects such as enhanced job lifecycle management, GPU virtualization, multi-cluster scheduling, and hierarchical resource queues, all contributing to a more robust and efficient platform for managing demanding AI/ML and HPC workloads on Kubernetes. The talk underscores Volcano's role as a CNCF incubation project, widely adopted and continuously evolving to meet the challenges of next-generation AI infrastructure.

Background

▶ Watch: Introduction to Cloud Native AI and Volcano Project (0:00)

The journey of Volcano began as kube-batch, a sub-project within the Kubernetes scheduling special interest group (SIG) back in 2015. The initial motivation stemmed from the realization that traditional Kubernetes scheduling, which primarily focuses on pod-by-pod allocation, was insufficient for the complex demands of batch workloads and high-performance computing (HPC). These workloads often require workload-awareness, gang scheduling, efficient queueing mechanisms, and fair sharing among multiple users, necessitating a more sophisticated approach than the default scheduler could provide. This led to the expansion of kube-batch into a standalone project, eventually becoming Volcano, now an incubation-level CNCF project.

Over the years, Volcano has grown to support a diverse range of workloads, including mainstream AI frameworks, big data processing, and HPC applications. The project's evolution has been driven by two key trends in cloud-native AI infrastructure. Firstly, at the resource layer, the focus has shifted from in-node topology awareness (like NUMA and feature discovery) to more critical inter-node network topology. This shift is crucial for supporting heterogeneous AI clusters from various hardware vendors, where network latency and bandwidth between nodes dramatically impact distributed training performance. Secondly, at the infrastructure management layer, the increasing complexity of workloads—especially distributed training and distributed inference—demands easier ways to map these deployment patterns to optimal underlying network topologies. Volcano's mission is to provide an abstraction layer that simplifies this complexity, allowing data scientists to focus on their models rather than intricate infrastructure configurations, while still achieving peak performance.

Key Findings

▶ Watch: Volcano Project: History, scope, and incubation status (4:20)

The talk presented several key findings and advancements within the Volcano project, all aimed at enhancing the efficiency, performance, and manageability of AI/ML workloads on Kubernetes:

1. HyperNode API for Network Topology Abstraction: The introduction of the HyperNode API provides a novel way to abstract and represent underlying network topologies within an AI cluster. This API defines a group of nodes with similar network performance characteristics, enabling a clearer semantic understanding of inter-node connectivity and facilitating more intelligent scheduling decisions. It supports nested structures, allowing for multi-layered representation of network domains.
2. Network Topology-Aware Scheduling: Building upon the HyperNode API, Volcano now offers network topology-aware scheduling. This allows users to specify network topology preferences for their workloads, ensuring that highly communicative pods are co-located on nodes with optimal network connectivity (e.g., within the same HyperNode or a specific tier of network connection). This significantly reduces communication latency and boosts distributed training performance.
3. Enhanced Job Lifecycle Management and Fault Recovery: Volcano has improved its job lifecycle management capabilities with multi-layered retry policies. Instead of restarting an entire job upon failure, users can now configure policies to reset only the failed pod or task, drastically improving job execution efficiency. Timeout semantics further refine this by allowing predefined actions to be skipped if a pod recovers within a specified window.
4. GPU Virtualization: To address the high cost and often low utilization of GPUs, especially in AI inference scenarios, Volcano offers robust GPU virtualization functionality. This supports both NVIDIA vGPU and MIG mode, along with a unified API using vgpu_memory and vgpu_number to request fractional GPU resources, enabling multiple pods to share a single GPU card.
5. Multi-Cluster Scheduling (Volcano Global): Recognizing the trend of managing workloads across multiple clusters, Volcano has incubated the Volcano Global sub-project. This extends Volcano's scheduling capabilities to a multi-cluster environment, supporting features like queue priority scheduling, multi-tenant fair sharing, and job priority across distributed Kubernetes deployments.
6. Hierarchical Queues for Resource Management: Volcano introduces hierarchical queues, a more fine-grained and non-flat structure for managing resource allocation across different teams or departments. This is particularly critical for migrating big data workloads from platforms like Yarn to cloud-native environments, enabling sophisticated resource sharing and reclamation policies using fields like capability, deserved, and guaranteeing.
7. Unified Workload Scheduling: Volcano natively supports scheduling both batch jobs and microservices in a unified manner. It also facilitates colocation of online and offline jobs and dynamic resource oversubscription to optimize resource utilization while ensuring Service Level Objectives (SLOs) for online services are met.

Technical Deep Dive

▶ Watch: Overview of Volcano's core features and APIs (6:00)

The technical core of Volcano's advancements lies in its ability to understand and leverage the underlying infrastructure's topology, particularly network configurations, for intelligent workload placement.

HyperNode API and Network Topology-Aware Scheduling

The most significant new feature is the HyperNode API, designed to abstract the complex inter-node network topology in high-performance AI clusters. While previous efforts focused on in-node topology (like NUMA awareness), the increasing scale of distributed AI training and inference necessitates awareness of how nodes are connected. Hardware solutions like Nvidia DGX SuperPods, which group nodes with high-performance GPU networks (MVL link, MV switch, InfiniBand, RoCE), exemplify the need for this abstraction.

A HyperNode object defines a logical group of nodes that share similar network performance characteristics, typically meaning they are connected by high-bandwidth, low-latency links. The API supports a nested data structure, allowing for a hierarchical representation of the network. For instance, nodes directly connected to a single switch (e.g., "switch zero") can form a "tier one" HyperNode. Multiple such tier-one HyperNodes might then be connected by a higher-level switch (e.g., "switch one"), forming a "tier two" HyperNode. This tree-like structure allows users to define multiple network trees within their cluster, accommodating various GPU network setups.

Users can leverage this abstraction by specifying network topology preferences in their PodGroup configurations. A new field, networkTopology, allows users to define their desired connectivity. For example, highestTierAllowed: 2 would instruct the Volcano scheduler to place pods within HyperNodes that are connected at tier 2 or lower, ensuring that heavily communicating pods are kept within a domain of optimal network performance. This is particularly useful for parallelization strategies like tensor parallelism within a HyperNode and data parallelism across different HyperNodes.

The process for enabling this involves an auto-discovery functionality. A HyperNode controller, working with provider plugins, automatically discovers the underlying hardware accelerators and network configurations from different vendors. This information is then translated into HyperNode definitions, forming the topology trees within the cluster. This mechanism also collects health status information, such as connectivity issues between nodes via switches, which is crucial for large-scale training where cable or switch failures can disrupt jobs.

The advantages of this API-driven approach over simple node labels are substantial:

• Clear Semantics: The HyperNode API provides clear, unified semantics and fields, unlike arbitrary node labels whose keys and values can vary by provider.
• Simplified Configuration: Standardized API makes configuring scheduling constraints easier and reusable across workloads.
• Scalability and Granularity: The nested structure offers flexible granularity, allowing users to constrain workloads to spread across different tiers of HyperNodes, which is difficult with flat node labels.
• Unified Management: The API facilitates lifecycle and status management, providing a clear view of HyperNode composition and health status, which is absent with node labels.

Enhanced Job Lifecycle Management and Fault Recovery

For distributed AI training and HPC, job failures due to hardware or software issues are common. Volcano's enhanced job lifecycle management addresses this with multi-layered retry policies. Previously, a failure might necessitate restarting the entire job. Now, users can configure policies to:

• Reset only the failed pod: If a specific pod fails, only that pod is restarted.
• Reset one task: If a task (a group of pods) fails, only that task is reset.

This granular control significantly improves job execution efficiency by avoiding unnecessary restarts of an entire, potentially long-running, distributed job. Additionally, timeout semantics allow users to define a time window for recovery. If a failed pod or task recovers within this window, predefined recovery actions (like restarting) are skipped, further optimizing resource usage and reducing disruption.

GPU Virtualization

The high cost of GPUs and their often low utilization, particularly in AI inference scenarios, makes efficient sharing critical. Volcano's GPU virtualization capabilities enhance efficiency by supporting both NVIDIA vGPU and MIG mode. It provides a unified API through vgpu_memory and vgpu_number fields, allowing users to request fractional GPU resources. This enables multiple pods to effectively share a single GPU card, maximizing hardware utilization and reducing operational costs.

Multi-Cluster Scheduling with Volcano Global

As organizations increasingly adopt multi-cluster strategies for managing workloads, Volcano has extended its reach with the Volcano Global sub-project. This initiative brings Volcano's advanced scheduling capabilities to a distributed Kubernetes environment. Key features of Volcano Global include:

• Queue priority scheduling across multiple clusters in multi-tenant setups.
• Fair share scheduling to ensure equitable resource distribution.
• Job priority scheduling to prioritize critical workloads across the global fleet.

This allows users to leverage Volcano's intelligence for resource orchestration beyond the boundaries of a single Kubernetes cluster.

Hierarchical Queues for Resource Management

Resource management is a cornerstone of shared cloud-native environments. Volcano introduces hierarchical queues to address the limitations of flat queues, which are insufficient for managing resource sharing in complex organizational structures (e.g., different teams or departments). This is especially pertinent for migrating big data workloads from Yarn to Kubernetes, where sophisticated resource governance is required.

Each queue in Volcano is defined with three important fields:

• capability: A hard quota representing the maximum resources a queue can consume.
• deserved: An elastic quota, indicating resources that a queue is entitled to but can be reclaimed by other queues if not in use.
• guaranteeing: Reserved resources that cannot be shared or reclaimed by other queues, ensuring a baseline for critical workloads.

This hierarchical structure and these fields provide a powerful mechanism for fine-grained resource allocation, enabling dynamic and fair sharing while respecting organizational boundaries and priorities.

Unified Workload Scheduling and Dashboard

Volcano natively integrates with Kubernetes' default scheduling algorithms, allowing it to schedule both batch jobs and microservices in a unified manner. This flexibility enables strategies like colocating online and offline jobs and dynamic resource oversubscription, which optimizes resource utilization by leveraging idle resources from online services for offline batch processing, all while ensuring that Service Level Objectives (SLOs) for online jobs are met.

The project also provides a resource dashboard. The current version allows users to view jobs, pod groups, queues, and their resource usage. The upcoming version will enhance this further by supporting the creation, deletion, and updating of these resources directly from the dashboard, offering even more control and flexibility.

Demo / Proof of Concept

▶ Watch: Real-world example: NVIDIA DGX SuperPod and HyperNode motivation (8:00)

The talk did not feature a live demonstration of the Volcano features in action. Instead, the speakers illustrated the concepts with detailed architectural diagrams, API definitions, and configuration examples. They walked through how the HyperNode API defines network topologies, how users specify networkTopology preferences in PodGroup configurations, and the overall workflow of auto-discovery and scheduling. While a live demo was not part of this presentation, the comprehensive visual aids and explanations served to clarify the technical implementation and operational flow of these advanced scheduling capabilities.

Defensive Implications

▶ Watch: Detailed explanation of HyperNode API and nested structure (10:00)

While this talk focuses on performance and efficiency rather than security, the "defensive implications" can be reinterpreted as operational best practices and optimization strategies for organizations deploying AI/ML workloads on Kubernetes using Volcano. Adopting these features helps defend against common operational challenges, resource inefficiencies, and performance bottlenecks:

1. Mitigating Performance Bottlenecks with Network Topology Awareness: By leveraging the HyperNode API and network topology-aware scheduling, organizations can actively defend against performance degradation caused by suboptimal workload placement. Ensuring that highly communicative distributed training jobs are co-located on nodes with high-bandwidth, low-latency network connections (e.g., within the same HyperNode or tier) significantly reduces communication overhead, leading to faster training times and more efficient GPU utilization. This is a critical defense against underperforming AI infrastructure.
2. Enhancing Job Resiliency and Reducing Downtime: The improved job lifecycle management with multi-layered retry policies and timeout semantics directly defends against job failures causing complete restarts and prolonged downtime. By allowing granular recovery (resetting only failed pods or tasks), organizations can minimize the impact of transient issues, accelerate fault recovery, and ensure higher availability and completion rates for their critical AI/ML workloads.
3. Optimizing GPU Resource Utilization and Cost Management: GPU virtualization is a powerful defense against the high cost and underutilization of expensive GPU hardware. By enabling fractional GPU sharing through NVIDIA vGPU, MIG mode, and Volcano's unified API (vgpu_memory, vgpu_number), organizations can maximize the return on investment for their GPU clusters, reduce operational expenditure, and make AI inference more economically viable.
4. Enforcing Resource Governance and Fair Sharing: The implementation of hierarchical queues with capability, deserved, and guaranteeing fields provides a robust mechanism for resource governance. This defends against resource contention, "noisy neighbor" problems, and unfair allocation across different teams or projects. It ensures that critical workloads receive their guaranteed resources while allowing for flexible sharing of elastic quotas, preventing resource starvation and promoting a fair multi-tenant environment.
5. Scaling AI Workloads Securely and Efficiently Across Clusters: Volcano Global for multi-cluster scheduling extends these benefits to distributed environments, defending against the complexity and inefficiencies of managing AI workloads across disparate Kubernetes clusters. It enables consistent scheduling policies, fair sharing, and priority management across the entire infrastructure, crucial for large-scale AI deployments.
6. Streamlining Operations with Unified Scheduling and Observability: Volcano's ability to schedule both batch jobs and microservices, alongside the upcoming enhancements to its resource dashboard, defends against operational silos and complexity. A unified scheduling plane simplifies management, while improved observability helps administrators quickly identify and address resource issues, contributing to a more stable and efficient AI platform.

In essence, these features empower organizations to build more resilient, performant, and cost-effective cloud-native AI infrastructures, effectively defending against the operational challenges inherent in scaling advanced machine learning.

Key Takeaways

• HyperNode API is a game-changer for AI/ML performance: Volcano's new HyperNode API provides a critical abstraction layer for inter-node network topology, enabling intelligent placement of distributed AI workloads based on network performance characteristics like those found in Nvidia DGX SuperPods.
• Network Topology-Aware Scheduling boosts efficiency: By allowing users to define networkTopology preferences and highestTierAllowed in PodGroup configurations, Volcano can optimize workload placement to reduce communication latency and significantly accelerate distributed training and inference.
• Granular fault recovery enhances job resiliency: Multi-layered retry policies and timeout semantics for job lifecycle management reduce the impact of failures by allowing resets of individual pods or tasks, rather than entire jobs, improving overall job execution efficiency.
• GPU virtualization drives cost savings and utilization: Volcano's support for NVIDIA vGPU and MIG mode, along with the unified vgpu_memory and vgpu_number API, enables efficient sharing of expensive GPU resources, critical for optimizing costs in AI inference and training.
• Hierarchical queues provide powerful resource governance: The introduction of hierarchical queues with capability, deserved, and guaranteeing fields offers fine-grained resource management for multi-tenant environments, ensuring fair sharing and guaranteed resources for critical workloads.
• Volcano offers a unified and scalable platform: As a CNCF incubation project, Volcano provides a comprehensive solution for scheduling diverse workloads (batch jobs, microservices, HPC, AI/ML) across single or multiple clusters (via Volcano Global), with features like dynamic resource oversubscription and an evolving resource dashboard.

About the Speaker(s)

The talk was primarily delivered by Xuzheng Chang, also known as Kevin Juan, who stepped in to present on behalf of William Wang. Kevin Juan has a deep background in scheduling, having started contributing to upstream Kubernetes back in 2015. He has been involved in numerous sub-projects and currently serves on the Technical Oversight Committee (TOC) for the broader Kubernetes community. He is also a maintainer of the Volcano project, bringing extensive expertise in cloud-native scheduling to the presentation.

He was joined by Shu Jun, also known as Zir, who is also a maintainer of the Volcano project. Together, their combined experience and roles as maintainers of Volcano provided authoritative insights into the project's current state and future direction for high-performance AI and machine learning workloads on Kubernetes.