Data Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom... Hichem Kenniche

Data Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers

Speakers: Hichem Kenniche, Data Engineering & Machine Learning Engineer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=BkQRGsVBhkc

Overview

In this insightful KubeCon EU talk, Hichem Kenniche, a seasoned data and machine learning engineer, delves into the critical challenge of efficiently running batch workloads like ETL (Extract, Transform, Load), ELT (Extract, Load, Transform), and ML (Machine Learning) training and serving at scale on Kubernetes. Kenniche highlights a fundamental mismatch: while Kubernetes was initially designed for stateless applications, the data engineering and ML communities, migrating from older ecosystems like Hadoop, increasingly rely on it for stateful, resource-intensive batch processing. This talk argues that the default Kubernetes scheduler, while a marvel of engineering for its intended purpose, falls short in addressing the unique requirements of these batch workloads, particularly in multi-tenant environments with finite resources.

The core of Kenniche's presentation revolves around the adoption and evaluation of custom Kubernetes schedulers, specifically Volcano and YuniKorn. He shares practical insights from an end-user perspective, detailing how these specialized schedulers provide essential capabilities such as gang scheduling, application awareness, and priority-based resource management that are absent in the default scheduler. The overarching motivation for this optimization isn't just performance, but a pragmatic drive towards efficient infrastructure utilization, substantial cost reduction, and a significant decrease in energy consumption and carbon footprint, challenging the notion that cloud-native implies infinite, disposable resources.

Background

▶ Watch: Speaker's background and Kubernetes batch workload challenges (0:00)

Hichem Kenniche frames the problem from the perspective of a data and machine learning engineer, a domain historically rooted in environments like Hadoop, where frameworks such as Apache Spark and Ray thrive. The migration of these workloads to Kubernetes presented a new set of challenges. Kubernetes, born nine years ago, was predominantly envisioned as a platform for stateless microservices. Consequently, its default scheduler, while "amazing" and "an amazing piece of technology" for its design goals, was not built with the specific needs of batch processing in mind.

The limitations of the default scheduler become glaringly apparent when dealing with complex data pipelines and ML tasks. Kenniche elaborates on these deficiencies:

• Lack of Application Concept: The default scheduler treats all pods as individual units, failing to distinguish between different applications (e.g., a Spark application versus a Ray application, Trino, or Flink). It operates on a "one large queue" principle at the pod level, lacking awareness of job-level dependencies or requirements.
• Limited Scheduling Policies: Crucially, it lacks gang scheduling, a fundamental requirement for many batch jobs where all components (e.g., Spark driver and executors) must be scheduled and run concurrently for the application to proceed. Without it, a job might get allocated only half its required pods, leading to a deadlock situation where resources are consumed but no progress is made.
• Limited Multi-Tenancy Support: In environments where multiple teams (data engineers, ML engineers, data scientists, researchers) share a finite set of cluster resources, the default scheduler struggles with fair resource allocation, priority handling, and enforcing quotas. This can lead to resource contention and inefficient utilization.
• Absence of Fairness and Priority Handling: While the default scheduler offers basic FIFO (First-In, First-Out) or fair scheduling, it lacks the sophistication to manage complex priorities or hierarchical queues, which are vital for ensuring that critical, time-sensitive jobs are prioritized over less urgent ones.

This problem is not new. Kenniche highlights Kube-batch, an initiative from 2018, as the "ancestor" project that first attempted to bring batch processing capabilities to Kubernetes. Although now archived, Kube-batch laid the foundational concepts that shaped the development of subsequent custom schedulers, proving to be visionary in anticipating the future needs of the platform. At that time, Spark on Kubernetes was still experimental, underscoring how far the ecosystem has evolved. The motivation to address these limitations stems from a desire not just for performance, but for sustainable, efficient infrastructure that minimizes resource waste, echoing a broader industry push for reduced energy consumption and a smaller carbon footprint.

Key Findings

▶ Watch: Goal: Running ETL/ML batch workloads at scale on Kubernetes (2:00)

The central finding of Kenniche's talk is the significant superiority of custom schedulers, specifically Volcano and YuniKorn, over the default Kubernetes scheduler for managing batch workloads. Through real-world testing in a multi-tenant environment, the speaker demonstrated that these specialized schedulers are not merely incremental improvements but provide foundational capabilities essential for efficient and scalable data processing and machine learning on Kubernetes.

Key findings include:

• Enhanced Resource Utilization: Both Volcano and YuniKorn demonstrably improve cluster utilization by a factor of two to three compared to the default scheduler. This directly translates to substantial cost savings and a reduced environmental footprint.
• Critical Feature Enablement: Custom schedulers introduce essential features missing in the default scheduler, such as:
• Gang Scheduling: Ensuring all pods required for a specific job are scheduled simultaneously, preventing deadlocks and resource starvation.
• Application Awareness: Treating workloads as coherent applications rather than disparate pods, allowing for intelligent, job-level scheduling decisions.
• Multi-Tenancy Support: Facilitating the sharing of finite cluster resources among diverse user groups (data engineers, ML engineers, data scientists) through hierarchical queues, quotas, and priority management.
• Priority-Based Scheduling: Allowing critical jobs to preempt lower-priority tasks, ensuring timely completion of high-value workloads.
• Volcano's Strengths: As a CNCF-backed project, Volcano is presented as a more comprehensive and highly configurable scheduler. It excels in managing tight control over resources, supporting extensive resource allocation, and is particularly well-suited for massive multi-tenant clusters, HPC (High-Performance Computing), and workloads requiring specialized hardware like GPUs. Its broad scope means it covers a wider array of use cases, though this comes with increased configuration complexity.
• YuniKorn's Strengths: Backed by the Apache Software Foundation, YuniKorn draws inspiration from the traditional Yarn cluster manager, bringing Yarn-like features to Kubernetes. It simplifies the operational overhead by abstracting Kubernetes complexities and offers robust support for hierarchical queues and resource guarantees. A notable advantage highlighted by Kenniche is YuniKorn's user-friendly UI for visual management, although he notes that Volcano is also developing one.
• Seamless Kubernetes Integration: Both schedulers integrate smoothly with the existing Kubernetes ecosystem, avoiding the need for "weird stuff outside" the platform, which is crucial for operational simplicity and maintainability.
• Limitations for Interactive Workloads: Despite their strengths for batch processing, neither Volcano nor YuniKorn are ideally suited for interactive workloads, such as data scientists running code in Jupyter notebooks connected to Spark clusters.
• Observed "Weird Behaviors": During testing, some non-reproducible "weird behaviors" were noted. These included occasional starvation in Volcano and instances where the Spark driver would die due to preemption in YuniKorn, suggesting areas for further community investigation and refinement.

In essence, the talk powerfully advocates for the strategic adoption of custom schedulers as a non-negotiable step for organizations serious about optimizing their data and ML pipelines on Kubernetes, leading to significant operational and economic advantages.

Technical Deep Dive

▶ Watch: Real-world problem: Cyclist navigation platform driving custom scheduler need (4:00)

The technical exposition begins with the fundamental challenge of running Apache Spark (and by extension, other big data frameworks like Ray, Trino, or Flink) on Kubernetes. The standard approach involves a Spark operator, available in both Go and Java implementations, which manages the lifecycle of Spark applications as native Kubernetes resources. However, the core issue remains the underlying scheduler.

Kenniche meticulously details the shortcomings of the default Kubernetes scheduler. While acknowledging its sophisticated design, which includes phases like prefiltering and various algorithms for pod-to-node assignment, its fundamental limitation for batch workloads lies in its lack of application-level awareness. It perceives scheduling as a task of placing individual pods, not orchestrating entire multi-pod applications. This leads to:

• Single Pod-Level Queue: The scheduler operates primarily with "one large queue" for pods, making it oblivious to the interdependencies within a single Spark application that might require a driver pod and multiple executor pods to be available simultaneously.
• Absence of Gang Scheduling: A critical requirement for many batch jobs is gang scheduling, where all components of an application must be scheduled as a collective unit. Without it, a Spark application might get two out of four required executor pods, while another application also gets two, leading to a deadlock where neither application can proceed, and resources remain idle or partially utilized.
• Limited Multi-Tenancy and Priority: The default scheduler lacks robust mechanisms for enforcing resource quotas, managing hierarchical queues, or implementing sophisticated priority-based preemption across different teams or workloads in a multi-tenant cluster. This means a low-priority, long-running job could inadvertently starve a high-priority, time-sensitive ML training task.

The historical context is provided by Kube-batch, an initiative from 2018 that served as the "ancestor" for modern custom schedulers. Though now archived, Kube-batch was visionary in laying the groundwork for batch processing on Kubernetes, introducing concepts that were later refined and expanded upon.

This brings us to the two primary custom schedulers evaluated:

Volcano

Volcano is presented as a robust, CNCF-backed project specifically designed for batch scheduling. Its core capabilities address the default scheduler's limitations directly:

• Tight Resource Control and Quotas: Volcano offers granular control over resource allocation and enforces quotas, essential for fair resource distribution in multi-tenant environments.
• Extensive Resource Allocation: It can manage complex resource requirements, making it suitable for demanding workloads like HPC and AI/ML batch processing.
• Specialized Hardware Support: Crucially, Volcano is adapted for GPUs and other specialized hardware, a significant advantage given the cost and demand for these resources in ML. Kenniche identifies the efficient mutualization and slicing of GPUs as his "next nirvana."
• Multi-Tenant Clusters: It's built for massive multi-tenant clusters, allowing different business teams or research groups to share resources effectively.
• Queue Management: Best practices for Volcano include leveraging its queue management aspect, which requires careful organization and definition of queues to manage job flow and priorities. Users must specify precise resource requirements for their jobs and manage complex job dependencies, moving away from unstructured "spaghetti DAGs" often seen in tools like Airflow.

YuniKorn

YuniKorn is an Apache Software Foundation-backed project, whose name (Yarn unified with Kubernetes) hints at its lineage. It aims to bring Yarn-like features to Kubernetes:

• Yarn-like Features: YuniKorn incorporates familiar concepts from Yarn, including gang scheduling, hierarchical queues, and resource guarantees, making the transition smoother for users familiar with the Hadoop ecosystem.
• Abstracted Kubernetes Complexity: A key benefit is its ability to abstract away some of the underlying Kubernetes complexities, providing a smoother experience for users.
• Multi-Tenant and Dynamic Resource Needs: It is ideal for multi-tenant clusters and environments with dynamic resource requirements, supporting priority-based scheduling.
• Visual UI: YuniKorn historically offered a nice UI for visualizing cluster state and queue management, which was a differentiator, though Volcano is catching up in this regard.
• Best Practices: Similar to Volcano, YuniKorn requires forethought in configuring hierarchical queues and resource quotas to align with organizational priorities.

Experimental Setup and Evaluation

Kenniche's team conducted pragmatic, real-world testing with a finite set of resources (emphasizing a departure from the "infinite cloud" mindset) in a multi-tenant setup. This involved:

• Workload Simulation: Handling approximately 100 events per hour, triggering various jobs ranging from CPU-intensive joins and database writes to full ML model retraining tasks. The maximum job capacity tested was around 1000 jobs, beyond which the Kubernetes API server became a bottleneck.
• Failure Simulation: Node failures (simulating cloud provider spot instance kills) were manually injected to test the resilience and recovery capabilities of jobs under the custom schedulers. This highlights the importance of job resilience in dynamic cloud environments.

The evaluation used a subjective, pragmatic scale:

• Scale (Efficiency/Performance): 3 stars (best) to 1 star (okay).
• Complexity (Configuration): 3 sad faces (complex) to 1 (okay).

The comparative results indicated:

• Volcano's Complexity: Volcano was rated as more complex to configure, primarily because its broader scope and completeness allow it to cover a wider range of use cases.
• Resource Utilization: Volcano showed a slight edge in overall resource utilization.
• Fault Tolerance and Recovery: Both performed similarly, handling preemption (resources taken from a lower-priority job for a higher one) and node failures effectively.
• Scheduling Latency and Load Distribution: Both were rated similarly, showing moderate performance improvements over the default.
• Job Completion Time: Crucial for long-running batch jobs (which can span days), both schedulers significantly improved completion rates, making the minor differences in latency less critical.
• Interactive Workloads: Neither scheduler proved well-suited for interactive use cases, indicating a gap for these specific scenarios.
• "Weird Behaviors": Specific issues included "starvation" in Volcano and the Spark driver occasionally dying due to preemption in YuniKorn, which the team couldn't consistently reproduce but observed in real-world conditions.

In conclusion, the technical deep dive underscores that custom schedulers are not just an option but a necessity for organizations seeking to run complex, resource-intensive batch workloads efficiently and reliably on Kubernetes, transforming it into a truly suitable platform for data and ML engineering.

Demo / Proof of Concept

▶ Watch: Spark on Kubernetes: benefits for efficient resource utilization, cost reduction (6:00)

While the talk did not feature a live, interactive demonstration of the custom schedulers in action, Hichem Kenniche detailed a comprehensive experimental setup and the results of pragmatic, real-world testing. This effectively served as a proof of concept for the capabilities of Volcano and YuniKorn in a multi-tenant Kubernetes environment.

The setup involved simulating a real-world scenario where a platform for cyclist navigation, built on Kubernetes with Apache Spark, needed to process 100 events per hour. These events triggered diverse batch jobs, from CPU-intensive joins to full ML model retraining, requiring varying resource profiles. The team used a finite set of resources to mimic typical production constraints and specifically simulated node failures (e.g., spot instance preemption) to test the schedulers' resilience. The evaluation metrics, though subjective, were derived from real operational experience, providing a practical assessment of how each scheduler handled workload distribution, job completion, fault tolerance, and resource utilization under stress. This detailed account of their testing methodology and observed outcomes served as a robust, field-tested validation of the custom schedulers' effectiveness.

Defensive Implications

▶ Watch: Default Kubernetes scheduler: amazing, but not adapted for batch (7:40)

For organizations leveraging Kubernetes for data engineering, machine learning, and other batch-oriented workloads, Hichem Kenniche's talk provides critical insights with actionable defensive implications:

1. Mandatory Adoption of Custom Schedulers: Platform engineers and SRE teams responsible for Kubernetes clusters supporting data and ML workloads should consider the adoption of custom schedulers like Volcano or YuniKorn as a strategic imperative, not an optional enhancement. This is crucial for achieving optimal resource utilization (2-3x improvement), reducing operational costs, and minimizing the environmental footprint.
2. Rethink Job Dependencies and Workload Design: Data scientists and ML engineers must move beyond "spaghetti DAGs" and design their batch jobs with the capabilities of custom schedulers in mind. This means explicitly defining resource requirements, understanding job dependencies, and structuring workloads to leverage gang scheduling and priority-based execution, ensuring that interdependent tasks can execute efficiently.
3. Implement Robust Multi-Tenancy Management: For multi-tenant clusters, configure hierarchical queues and enforce strict resource quotas using the custom schedulers. This ensures fair resource allocation across different teams (data engineering, ML, data science) and prevents any single team or low-priority job from monopolizing or starving critical resources.
4. Prioritize Critical Workloads: Utilize the priority mechanisms offered by custom schedulers to ensure that high-value, time-sensitive tasks (e.g., critical model retraining, real-time data ingestion pipelines) receive preferential treatment, potentially preempting lower-priority jobs when resources are scarce. This requires clear organizational policies on workload prioritization.
5. Optimize for Specialized Hardware: For workloads relying on expensive specialized hardware like GPUs, leverage the advanced scheduling capabilities of Volcano (and increasingly YuniKorn) to enable efficient GPU mutualization and slicing. This maximizes the return on investment for these costly resources and allows for more granular control over their allocation.
6. Enhance Monitoring and Observability: Implement comprehensive monitoring for custom scheduler metrics, including scheduling latency, queue depth, job completion rates, and resource utilization per queue or application. Be vigilant for "weird behaviors" like job starvation (Volcano) or unexpected driver deaths due to preemption (YuniKorn), and establish alerting mechanisms to address these promptly.
7. Test for Resilience and Preemption: Regularly simulate node failures (e.g., spot instance preemption) and test how your batch jobs and the custom schedulers handle these disruptions. Validate that jobs are resilient, can restart correctly, and that preemption mechanisms behave as expected without leading to unrecoverable states or prolonged job delays.
8. Evaluate Interactive Workload Strategy: Acknowledge that current custom schedulers are not ideal for interactive workloads. Organizations should evaluate alternative strategies for interactive data exploration and ML experimentation, which might involve separate clusters or different scheduling approaches, to avoid compromising the efficiency of batch processing.

By proactively addressing these areas, organizations can transform Kubernetes into a highly efficient, cost-effective, and resilient platform for their most demanding data and machine learning workloads.

Key Takeaways

• Custom Schedulers are Essential for Batch Workloads: The default Kubernetes scheduler is insufficient for efficient batch processing (ETL, ELT, ML training/serving). Custom schedulers like Volcano and YuniKorn are critical for optimizing these demanding workloads.
• Significant Resource Utilization Gains: Adopting custom schedulers can improve cluster utilization by a factor of 2-3, leading to substantial reductions in infrastructure costs, energy consumption, and carbon footprint.
• Enable Critical Batch Features: These schedulers provide vital capabilities missing in the default scheduler, including gang scheduling, application awareness, multi-tenancy support, and priority-based scheduling, which are fundamental for reliable and efficient batch job execution.
• Volcano for Comprehensive Control, YuniKorn for Yarn-like Features: Volcano (CNCF-backed) offers extensive configuration, broad use case coverage (including HPC and GPUs), and tight resource control. YuniKorn (Apache Software Foundation-backed) provides Yarn-like features, hierarchical queues, and a user-friendly UI, abstracting some Kubernetes complexities.
• Rethink Job Design and Resource Management: Effective utilization of custom schedulers requires careful planning of job dependencies, precise specification of resource requirements, and thoughtful configuration of hierarchical queues and resource quotas, especially in multi-tenant environments.
• Limitations for Interactive Workloads: While highly effective for batch processing, neither Volcano nor YuniKorn are currently well-suited for interactive workloads, suggesting a need for alternative strategies for such use cases.

About the Speaker(s)

Hichem Kenniche is a seasoned professional with a background in data engineering and machine learning engineering. He describes himself as an "end user" and "practitioner" who has navigated the transition from the traditional Hadoop ecosystem to Kubernetes. Kenniche is passionate about leveraging technology to build efficient infrastructure, reduce energy utilization, and minimize the carbon footprint of computing. His insights are drawn from hands-on experience in building open-source platforms, such as a navigation system for cyclists, which necessitates robust and efficient batch processing capabilities on Kubernetes.