SIG Scheduling Intro & Updates - Maciej Skoczeń, Google; Kensei Nakada, Tetrate.io

SIG Scheduling Intro & Updates

Speakers: Maciej Skoczeń, Google; Kensei Nakada, Tetrate.io

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=ZIk_EqI8rVA

Overview

This session, presented by Maciej Skoczeń from Google and Kensei Nakada from Tetrate.io, offered a comprehensive overview and update on the Kubernetes SIG Scheduling. The talk delved into the fundamental architecture of the Kubernetes scheduler, the core component responsible for intelligently placing pods onto nodes within a cluster. Beyond the foundational concepts, the speakers highlighted significant recent enhancements focused primarily on improving scheduling performance and introducing advanced resource management capabilities.

The importance of this talk lies in its direct impact on the efficiency, scalability, and stability of Kubernetes clusters. As clusters grow in size and complexity, the performance of the scheduler becomes a critical bottleneck. The presented updates, such as Queuing Hints and Async Preemption, directly address these performance challenges, aiming to reduce unnecessary processing and optimize resource utilization. Furthermore, the introduction of the Device Assignment (DA) Array signifies a crucial step towards more sophisticated hardware scheduling, enabling Kubernetes to better support specialized workloads requiring specific device configurations.

For anyone operating or developing on Kubernetes, understanding these updates is paramount. The session provided valuable insights into how the scheduler is evolving to meet the demands of modern cloud-native applications, offering practical implications for cluster design, workload deployment, and overall operational efficiency. The discussion also extended to key sub-projects, showcasing the broader ecosystem of tools designed to enhance Kubernetes scheduling and resource management.

Background

▶ Watch: SIG Scheduling introduction and Kube-scheduler's role (0:00)

The Kubernetes scheduler is a crucial control plane component that watches for newly created pods with no assigned node, and selects a node for them to run on. This decision-making process is complex, involving numerous factors such as resource requirements (CPU, memory), pod affinity and anti-affinity rules, node affinity, and pod topology spread constraints. Each of these considerations is implemented as a "plugin" within the scheduler's sophisticated scheduling framework.

The framework operates through several extension points, with Filter and Score being two major ones during the scheduling cycle. At the Filter extension point, plugins reject nodes that cannot accommodate a pod (e.g., a node lacking sufficient resources, handled by the ResourceFit plugin, or not matching required labels, handled by NodeAffinity). Nodes that pass all filters then proceed to the Score extension point, where plugins assign a preference score to each remaining node. For instance, the ImageLocality plugin might give a higher score to nodes that already have the required container image cached, reducing pull times. The node with the highest total score is ultimately chosen.

Once a node is selected, the pod enters the binding cycle, where the scheduler updates the pod's nodeName field in the Kubernetes API. This update signals to the kubelet on the chosen node to start the pod. A key architectural design choice is the asynchronous nature of the binding cycle, which decouples the decision-making from the API calls, improving efficiency. The scheduler also maintains a scheduling queue to manage pending pods, prioritizing them based on factors like priority class. Historically, a significant challenge for the Kubernetes scheduler has been maintaining high throughput, especially in large and dynamic clusters. Performance improvements have been a continuous focus, as inefficient scheduling can lead to pods remaining in a pending state for extended periods, impacting application availability and resource utilization.

Key Findings

▶ Watch: Understanding the complete Kubernetes scheduling framework architecture (4:00)

The talk highlighted several critical enhancements and updates within the Kubernetes scheduling ecosystem, primarily focusing on performance, resource management, and specialized hardware support introduced in recent Kubernetes versions, particularly leading up to 1.33.

1. Queuing Hints: A performance-driven feature designed to reduce unnecessary scheduling retries for unschedulable pods. By allowing plugins to provide "hints" about specific cluster events that could resolve a pod's unschedulability (e.g., a new node being created that is actually large enough for the pod), the scheduler can make more intelligent decisions about when to re-queue and retry a pod, leading to increased scheduling throughput.

1. Async Preemption: This enhancement addresses the performance overhead associated with preemption, where higher-priority pods evict lower-priority ones to secure a node. Previously, the preemption process, which involves API calls to delete pods, blocked the scheduling cycle. Async Preemption decouples these API calls, allowing them to run asynchronously, thereby preventing them from negatively impacting scheduling throughput.

1. Pop Pods from Backoff Queue when Active is Empty: This feature optimizes scheduler utilization. The scheduler's active queue holds pods ready for immediate scheduling, while the backoff queue holds pods that failed scheduling and are waiting for a penalty period to expire. When the active queue is empty, the scheduler previously idled. The new mechanism allows the scheduler to proactively pull pods from the backoff queue (after pre-queue checks) when the active queue is vacant, ensuring continuous operation and reducing idle time.

1. Device Assignment (DA) Array: This introduces a more flexible and expressive way to request specialized hardware resources. Beyond simple CPU/memory/GPU requests, DA allows pods to specify complex needs, such as fractions of devices or specific device types, through Resource Claims. Nodes, in turn, can expose Resource Slices that match these claims. Key features in 1.33 supporting DA include:

• Partitionable Devices: Allowing pods to claim a portion of a larger device, potentially spanning multiple nodes (multi-host).
• Prioritized Alternatives: Enabling pods to specify a preferred order of alternative devices if the primary request cannot be met.
• Device Taints and Tolerations: Extending the concept of node taints to devices, allowing for eviction of pods using a device if a taint is applied.

1. Other Kubernetes 1.33 Updates:

• Graduation of match label keys in pod affinity and anti-affinity to General Availability (GA).
• Graduation of node inclusion policy in pod topology spread to GA.
• Significant improvements in scheduling throughput, particularly for pods using inter-pod affinity, pod topology spread filtering, and certain preemption scenarios, showing an approximate 20% improvement in large clusters.

1. Sub-project Updates: The talk also covered updates in related SIG Scheduling sub-projects:

• Kueue: A component for managing quotas and job consumption, now with built-in integration for leader-worker sets, multi-queue support for Kubeflow and Ray clusters, and fair sharing of borrowable resources.
• Descheduler: A tool that evicts pods from nodes that no longer meet policy requirements (e.g., topology spreading, anti-affinity violations). It now exposes metrics for better observability and policy guidance.
• kube-scheduler-simulator: A tool for testing custom scheduler plugins and configurations in a fake cluster environment, now capable of connecting to a real cluster via kubeconfig to download existing resources, enabling more realistic simulation.

These findings collectively underscore a strong commitment within SIG Scheduling to enhance the performance, flexibility, and manageability of Kubernetes resource allocation, addressing both general cluster efficiency and specialized workload requirements.

Technical Deep Dive

▶ Watch: Decoupling scheduling decisions and binding API calls (5:40)

The enhancements discussed represent sophisticated architectural and algorithmic improvements to the Kubernetes scheduler. Each update targets specific bottlenecks or limitations, contributing to a more robust and efficient scheduling system.

Queuing Hints: Intelligent Retry Mechanisms

The conventional approach to re-queuing unschedulable pods involved a broad re-evaluation triggered by general cluster events. When a pod was rejected by a particular plugin (e.g., ResourceFit), the scheduler would note this and, upon detecting relevant cluster events (like a new node being added or an existing pod being removed), would re-queue the unschedulable pod for another attempt. The problem with this approach was its lack of specificity: not every "node added" event is relevant to every unschedulable pod. A new, small node might be added, but it wouldn't resolve a large pod's resource requirements.

Queuing Hints address this inefficiency. They allow individual plugins to provide more granular information to the scheduling queue about what specific conditions would make a pod schedulable again. For example, the ResourceFit plugin, when rejecting a pod, can now attach a hint specifying that the pod only needs to be retried if a new node is added and that new node has sufficient capacity for the pod's request. If a new node is added but doesn't meet this specific hint, the pod is not unnecessarily re-queued. This filtering of irrelevant events significantly reduces the number of futile scheduling retries, thereby improving the overall scheduling throughput. This is a prime example of pushing intelligence closer to the source of the decision (the plugin that knows why it rejected the pod) to optimize the global system.

Async Preemption: Decoupling API Calls for Performance

Preemption is a critical feature that ensures higher-priority workloads can always find a place in the cluster, even if it means evicting lower-priority pods. However, the preemption process previously created a performance bottleneck. When the scheduler decided to preempt pods on a node to make space for a high-priority pod, it would initiate API calls to delete those lower-priority pods. Crucially, the scheduling cycle would wait for these API calls to complete before starting the next cycle. Since API calls are network-bound and can introduce latency, this synchronous waiting directly impacted the scheduler's throughput, especially in scenarios with frequent preemption.

Async Preemption tackles this by making the API calls non-blocking. When the scheduler identifies a node for preemption, it now makes a "reservation" for the high-priority pod on that node and then immediately proceeds to the next scheduling cycle without waiting for the actual pod deletions to finish. The API calls for deleting the preempted pods are then executed asynchronously in the background. Subsequent scheduling cycles are designed to be aware of these ongoing preemption processes and factor in the reserved capacity when evaluating nodes. This decoupling mirrors the asynchronous nature of the binding cycle and significantly improves the scheduler's responsiveness and overall throughput by preventing API call latency from stalling the core scheduling logic.

Pop Pods from Backoff Queue when Active is Empty: Maximizing Scheduler Utilization

The Kubernetes scheduler manages pods across several internal queues:

• Active Queue: Contains pods ready to be scheduled immediately.
• Backoff Queue: Holds pods that have recently failed scheduling and are undergoing a "backoff penalty" period to prevent them from overwhelming the scheduler with repeated, doomed attempts. The backoff duration scales with the number of failed attempts.
• Unschedulable Pods: A map-like structure for pods that are currently deemed unschedulable (e.g., no node meets their requirements) and are awaiting specific cluster events to potentially become schedulable again.

Previously, if the active queue became empty, the scheduler would idle, even if there were pods in the backoff queue that could potentially be scheduled if their backoff period ended. The scheduler would only process the backoff queue periodically, moving pods to the active queue after their penalty expired. This led to periods of underutilization for the scheduler.

The new feature introduces a mechanism to address this. When the active queue is empty, the scheduler can now proactively "pop" pods from the backoff queue for immediate consideration. To support this, several architectural changes were necessary:

1. Pre-Enqueue (PNQ) plugins moved: The checks performed by PNQ plugins (which might reject a pod even before it enters the active queue) were moved to occur before a pod is added to the backoff queue. This ensures that any pod popped from the backoff queue has already passed these initial checks, making the pop operation itself as performant as possible.
2. Backoff Queue Ordering: The backoff queue's internal ordering function was modified to prioritize pods by their priority, similar to the active queue. This ensures that if the scheduler pulls from the backoff queue, it will always prioritize the highest-priority pods first, reducing the likelihood of future preemptions.
3. Error Handling: Pods in the backoff queue due to transient API errors (e.g., during binding) are still handled separately to prevent them from continuously exhausting the API server.

This change significantly improves scheduler utilization by minimizing idle periods, allowing it to process pods more continuously and efficiently.

Device Assignment (DA) Array: Advanced Hardware Scheduling

Traditional Kubernetes resource requests allow specifying CPU, memory, and simple device counts (like nvidia.com/gpu: 1). However, this model falls short for complex hardware scenarios. Many modern accelerators or specialized devices might require:

• Fractional allocation: A pod needing only 25% of a GPU's capacity.
• Specific device properties: A pod needing a GPU from a particular vendor, with a minimum amount of VRAM, or with specific network connectivity.
• Multi-host devices: A single logical device spanning multiple physical nodes.

The Device Assignment (DA) Array aims to provide a more flexible and expressive framework for these needs. Instead of simple resource requests, pods can now use Resource Claims to specify their exact hardware requirements. These claims are then matched against Resource Slices advertised by nodes, which describe the available device configurations.

Key features introduced in Kubernetes 1.33 related to DA include:

• Partitionable Devices: This allows a single physical device to be logically partitioned, and pods can claim a specific partition. This is particularly relevant for multi-host devices, where a pod might require a partition that spans parts of a device across different nodes.
• Prioritized Alternatives: Pods can define a list of preferred devices or device configurations. If the most preferred option isn't available, the scheduler can attempt to fulfill the request with a less preferred, but still acceptable, alternative. This improves scheduling flexibility and resource utilization by allowing for graceful degradation.
• Device Taints and Tolerations: Extending the familiar concept from nodes, this allows administrators to "taint" specific devices. Pods must then declare "tolerations" for these device taints to be scheduled on nodes utilizing them. This enables advanced management, such as isolating certain workloads to specific hardware or performing maintenance by tainting a device and then "evicting" pods that don't tolerate the taint.

Future challenges for DA involve dynamic device attachment and cross-node dependencies that might require more than the current pod-by-pod scheduling approach.

Demo / Proof of Concept

▶ Watch: Recent scheduler updates focusing on performance improvements (6:50)

While a live, step-by-step demonstration of each new feature (like Queuing Hints or Async Preemption) was not conducted during the presentation, the speakers extensively highlighted the kube-scheduler-simulator as a crucial tool for understanding, testing, and validating scheduler behavior and custom configurations.

The kube-scheduler-simulator provides a powerful environment for users to:

• Test Custom Plugins: Developers can implement their own scheduler plugins and test their functionality without impacting a production or even a staging cluster.
• Tweak Configurations: Experiment with different scheduler profiles, weights for scoring plugins, or other configuration parameters to optimize scheduling decisions for specific workloads or cluster characteristics.
• Visualize Scheduling Steps: The simulator features a user-friendly UI that graphically illustrates each stage of the scheduling cycle. This includes showing which nodes were rejected by which filter plugins, and the scores assigned by various scoring plugins, providing invaluable insight into the scheduler's decision-making process.

A significant new capability of the kube-scheduler-simulator is its ability to connect to a real cluster using a standard kubeconfig file. This eliminates the need for users to manually recreate their cluster's resources (nodes, pods, etc.) within the simulator. Instead, the simulator can download the actual state of a live cluster, allowing users to test tweaked scheduler configurations against a realistic, existing environment. This feature bridges the gap between theoretical testing and practical application, making it easier for operators to validate changes before deployment.

Although not a direct demonstration of the new features in action, the simulator itself serves as a robust proof-of-concept for how users can explore and understand the intricate workings of the Kubernetes scheduler, including the impact of new features and custom logic.

Defensive Implications

▶ Watch: Queuing hints: resolving unschedulable pods with cluster events (8:00)

While this talk focuses on performance and feature enhancements rather than security vulnerabilities, the "defensive implications" for cluster operators revolve around optimizing cluster resilience, resource utilization, and operational stability. Leveraging these new scheduler capabilities can significantly strengthen a Kubernetes environment against common operational challenges.

1. Improved Cluster Stability and Throughput: Queuing Hints and Async Preemption directly address performance bottlenecks. By reducing unnecessary scheduler load and preventing API calls from blocking the scheduling cycle, these features contribute to a more responsive and stable control plane. For defenders, this means a cluster that is less prone to "scheduler storm" scenarios, where a large influx of pods can overwhelm the scheduler, leading to prolonged pending states. A more performant scheduler can handle higher churn rates and scale more effectively, preventing resource starvation for critical workloads.

1. Efficient Resource Utilization: The "Pop Pods from Backoff Queue" feature ensures that scheduler resources are utilized continuously, minimizing idle time. This contributes to overall cluster efficiency, as pods are scheduled more promptly, leading to faster application startup times and better utilization of available node capacity. From a defensive perspective, efficient resource allocation means less wasted compute, better cost management, and a more predictable environment for deploying applications.

1. Advanced Hardware Management: The Device Assignment (DA) Array features (partitionable devices, prioritized alternatives, device taints/tolerations) provide powerful new tools for managing specialized hardware. Defenders can use these to:

• Isolate Sensitive Workloads: Utilize device taints to ensure that highly sensitive or regulated workloads are confined to specific, hardened hardware devices, preventing co-location with less trusted applications.
• Optimize Hardware Utilization: Leverage partitionable devices to maximize the use of expensive accelerators, allowing multiple smaller workloads to share a single device without resource contention. Prioritized alternatives ensure that workloads can still run even if their absolute preferred hardware is unavailable, increasing overall job completion rates.
• Graceful Maintenance: Device taints can be used to drain specific devices for maintenance, ensuring that pods are gracefully evicted and rescheduled, minimizing disruption.

1. Enhanced Policy Enforcement with Sub-projects:

• Kueue empowers cluster administrators to implement robust quota management and fair sharing policies for workloads, preventing resource monopolization and ensuring that critical jobs receive their allocated share. This is crucial for multi-tenant environments or clusters with diverse workload priorities.
• Descheduler acts as a continuous enforcement mechanism for scheduling policies post-placement. It can evict pods that violate topology spread constraints or anti-affinity rules due to changes in cluster state, proactively maintaining the desired cluster topology and preventing performance degradation or compliance issues.
• The kube-scheduler-simulator is an invaluable tool for testing and validating any changes to scheduler configuration or custom plugins in a safe, isolated environment. This "shift-left" approach to testing helps identify potential misconfigurations or performance regressions before they impact a production cluster, significantly reducing operational risk.

By understanding and strategically implementing these advancements, cluster operators can build more resilient, performant, and cost-effective Kubernetes infrastructures, effectively "defending" against common operational pitfalls and ensuring optimal workload execution.

Key Takeaways

• Performance is Paramount: Recent Kubernetes scheduler enhancements, like Queuing Hints and Async Preemption, are primarily focused on improving scheduling throughput and reducing latency by optimizing internal processes and decoupling blocking API calls.
• Smarter Pod Retries: Queuing Hints allow scheduler plugins to provide specific conditions for re-queuing unschedulable pods, drastically reducing unnecessary retries and improving efficiency.
• Continuous Scheduler Utilization: The new mechanism to pop pods from the Backoff Queue when the Active Queue is empty ensures the scheduler is always working, minimizing idle time and speeding up pod placement.
• Advanced Hardware Scheduling: The Device Assignment (DA) Array introduces a powerful framework for expressing complex hardware requirements, including partitionable devices, prioritized alternatives, and device taints/tolerations, moving beyond simple resource counts.
• Ecosystem of Tools: Sub-projects like Kueue (quota management), Descheduler (policy enforcement), and kube-scheduler-simulator (testing and visualization) are crucial for comprehensive Kubernetes resource management and operational excellence.
• Significant Throughput Gains: Kubernetes 1.33 saw up to a 20% improvement in scheduling throughput for specific scenarios involving inter-pod affinity, topology spread, and preemption in large clusters.

About the Speaker(s)

Maciej Skoczeń is a software engineer at Google, where he is actively involved in the AI training team. His work focuses on the critical area of AI infrastructure, suggesting a deep understanding of high-performance computing and specialized resource requirements within large-scale Kubernetes environments. His contributions to the SIG Scheduling community are instrumental in shaping the future of Kubernetes resource management for demanding workloads.

Kensei Nakada is a software engineer at Tetrate.io, a company known for its contributions to the service mesh ecosystem. His involvement in "service mesh stuff" indicates expertise in distributed systems, networking, and the operational challenges of deploying and managing complex microservices architectures on Kubernetes. His perspective brings valuable real-world experience from a major cloud-native vendor to the SIG Scheduling discussions.