A Practical Guide To Benchmarking AI and GPU Workloads in Kubernetes - Yuan Chen & Chen Wang

A Practical Guide To Benchmarking AI and GPU Workloads in Kubernetes

Speakers: Yuan Chen, Senior Solutions Architect, Nvidia; Chen Wang, Senior Research Scientist, IBM Research

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=OnzgoBf7dUE

Overview

This talk, presented by Yuan Chen from Nvidia and Chen Wang from IBM Research, offers a comprehensive guide to benchmarking AI and GPU-intensive workloads within Kubernetes environments. The session dives into practical methodologies and tools essential for understanding, optimizing, and ensuring the efficient operation of modern AI inference and generative AI (GenAI) applications. It addresses the critical need for robust benchmarking in the rapidly evolving landscape of AI, where performance, scalability, and resource utilization are paramount.

The speakers highlight how proper benchmarking can unveil bottlenecks, inform resource allocation strategies, and validate optimizations across diverse hardware and software stacks. This presentation is particularly relevant for AI/ML engineers, DevOps professionals, and infrastructure architects who are tasked with deploying and managing AI models at scale on Kubernetes, providing them with actionable insights and tools to achieve consistent and reproducible performance.

The talk underscores the importance of moving beyond basic performance metrics to capture nuanced behaviors of large language models (LLMs) and other complex AI architectures, especially concerning their unique demands on GPU resources and distributed systems. By demonstrating both general-purpose and specialized benchmarking frameworks, Chen and Wang equip attendees with the knowledge to make data-driven decisions for their AI infrastructure.

Background

▶ Watch: Welcome and session overview (0:00)

The proliferation of AI, particularly in areas like large language models and generative AI, has introduced significant challenges for infrastructure management and performance optimization. Deploying these computationally intensive workloads, often relying heavily on Graphics Processing Units (GPUs), within a containerized and orchestrated environment like Kubernetes requires specialized tools and methodologies. Traditional benchmarking approaches, while useful for general applications, often fall short in capturing the specific performance characteristics and resource demands of AI inference.

Prior to the sophisticated tools discussed, understanding the interplay between AI models, GPU hardware, and Kubernetes scheduling mechanisms was often a manual and iterative process. Key challenges include:

• Diverse Model Formats and Frameworks: AI models come in various formats (PyTorch, TensorFlow, ONNX, VLM) and are developed using different frameworks, requiring flexible serving solutions.
• GPU Resource Management: Efficiently sharing and allocating GPU resources (e.g., using time-slicing or Multi-Process Service - MPS) among multiple AI workloads is crucial for cost-effectiveness and performance, but requires careful benchmarking to understand trade-offs.
• Dynamic Workload Patterns: AI inference requests can vary significantly in size, concurrency, and arrival rate, necessitating load generation tools that can simulate realistic scenarios.
• LLM-Specific Metrics: For generative AI, metrics like "time to first token" and "inter-token latency" are more indicative of user experience than traditional throughput or end-to-end latency.
• Reproducibility and Consistency: Benchmarking in a distributed, dynamic environment like Kubernetes demands tools that can ensure consistent and reproducible results across different infrastructure configurations.

The Nvidia Triton Inference Server emerged as a popular open-source solution to address the serving aspect, providing a highly optimized, cloud-native inference server capable of handling multiple models and frameworks. However, while Triton excels at serving, the ecosystem around robust, Kubernetes-native benchmarking for these complex AI systems, especially GenAI, still presented gaps that the tools presented in this talk aim to fill. The need for a "code-first" approach to automate Kubernetes deployments for benchmarking, as highlighted by FMPERF, is a direct response to the complexity and manual overhead associated with configuring and running benchmarks in these environments.

Key Findings

▶ Watch: Triton setup: creating model repositories (2:00)

The talk presents several key findings and contributions, primarily centered around practical tools and methodologies for benchmarking AI and GPU workloads in Kubernetes:

1. Triton Inference Server as a Benchmarking Foundation: The Nvidia Triton Inference Server is established as a highly versatile and performant open-source tool for deploying and serving a wide array of AI models (PyTorch, ONNX, VLM) across x86 and ARM architectures. Its native support for HTTP and gRPC protocols, coupled with its client libraries and the Performance Analyzer tool, makes it an ideal platform for general AI inference benchmarking.
2. Specialized Benchmarking for Generative AI: The introduction of GI PF (Generative AI Performance Framework) within the Triton ecosystem, and more significantly, the FMPERF (Foundation Model Performance) library, addresses the unique benchmarking needs of large language models and generative AI. These tools provide LLM-specific metrics such as "time to first token" and "inter-token latency," which are crucial for evaluating user experience and model efficiency.
3. Kubernetes-Native Automation with FMPERF: FMPERF is highlighted as a significant open-source contribution (donated to the serving working group by IBM Research). It provides a simple yet powerful Python API for automating the deployment of inference servers (like vLLM, TGI server/TJS) and load testing jobs within Kubernetes or OpenShift. This "code-first" approach greatly simplifies the benchmarking process, ensuring consistency and reproducibility across diverse infrastructures.
4. Essential Monitoring Tools: The talk emphasizes the critical role of GPU monitoring tools like Nvidia SMI (System Management Interface) and the open-source GPU stats in understanding real-time GPU utilization, temperature, and memory consumption during benchmark runs. These tools provide vital insights into hardware behavior that complement software-level performance metrics.
5. Insights into GPU Sharing Strategies: Referencing a previous KubeCon talk by Kevin Cruz and Yuan Chen, the speakers underscore how benchmarking with Triton has provided valuable insights into the trade-offs of different GPU sharing strategies (e.g., time-slicing vs. MPS - Multi-Process Service) for varying AI workloads. This demonstrates the practical application of these benchmarking tools in optimizing resource allocation.
6. Focus on Reproducibility and End-to-End Solutions: A recurring theme is the need for consistent and reproducible benchmarking results, especially in dynamic Kubernetes environments. The call for more realistic, end-to-end performance analysis, encompassing data processing, inference, and post-processing, points towards the future direction of AI benchmarking.
7. Community Collaboration and Open Source: The speakers advocate for open-source tools and community collaboration, exemplified by FMPERF's donation to the serving working group, to enhance and develop new features for AI benchmarking, emphasizing that all tools discussed are publicly available.

Technical Deep Dive

▶ Watch: Deploying Triton server with YAML configuration (4:00)

The core of the talk revolves around two primary frameworks for benchmarking AI and GPU workloads in Kubernetes: the Nvidia Triton Inference Server with its accompanying tools, and the FMPERF Python library.

Triton Inference Server for General AI Benchmarking

The Nvidia Triton Inference Server is presented as a versatile, open-source solution for serving AI models. It supports a wide range of models, including those from PyTorch, TensorFlow, ONNX, and VLM (Vision Language Models). Triton is compatible with both x86 and ARM architectures and offers client libraries for various environments, including edge and mobile devices. Nvidia recently announced an advanced version called Dynamo, which extends Triton with advanced features for improved scalability and performance.

Setting up Triton for benchmarking involves three key steps:

1. Model Repository Creation: Users must create a model repository, which is a structured directory containing the AI models. Each model resides in its own subdirectory, identified by its name (e.g., text-detection). Within each model directory, a config.pbtxt file defines model-specific configurations, and multiple versions of a model can be stored (e.g., version1). Triton supports various model formats, allowing flexibility in what can be deployed. Models can be downloaded from online repositories and stored in a persistent volume.
2. Triton Server Deployment: The Triton server itself is typically deployed as a containerized application within Kubernetes. A standard Kubernetes YAML file defines the deployment, specifying the container image (often from Nvidia's registry), exposed ports for HTTP and gRPC protocols, and crucially, mounting the model repository into the container (e.g., /models). Services are then defined to expose these ports for external access.
3. Workload Generation and Performance Analysis: To benchmark the deployed Triton server, the Performance Analyzer tool is used. This client-side utility generates inference requests and sends them to the Triton server. It supports various modes, allowing users to specify concurrency levels, request rates (e.g., 200 requests per second), and measurement intervals. The Performance Analyzer collects detailed performance data, including:

• Total requests sent.
• Latency at different percentiles (e.g., 50th, 90th, 95th, 99th percentile).
• Throughput.
• Batch size performance.

For generative AI models, a newer workload generator called GI PF (Generative AI Performance Framework) is under active development. This specialized tool provides metrics critical for LLMs, such as time to first token and inter-token latency, offering deeper insights into the user experience of conversational AI.

Monitoring Tools

During benchmarking, real-time monitoring of GPU resources is essential. Two primary tools are recommended:

• Nvidia SMI (System Management Interface): A command-line utility for monitoring and managing Nvidia GPU devices. It provides comprehensive data on GPU utilization, temperature, memory usage, and other critical parameters for each GPU device (e.g., on an A100 node with eight GPUs).
• GPU stats: An open-source tool that offers real-time visualization of GPU temperature, utilization, and memory consumption, providing an immediate overview of resource behavior.

FMPERF for Kubernetes-Native GenAI Benchmarking

FMPERF (Foundation Model Performance) is a Python-based benchmarking library developed by IBM Research and open-sourced, now donated to the Kubernetes serving working group. Its primary goal is to simplify and automate the performance and energy efficiency evaluation of large language models and generative AI serving frameworks like vLLM and TGI server (including IBM's fork, TJS).

FMPERF's key advantages include:

• Simple Python API: It allows users to define and orchestrate complex benchmarking experiments with concise Python code, abstracting away the complexities of Kubernetes YAML definitions.
• Kubernetes Integration: FMPERF leverages the Kubernetes client library to automatically deploy inference servers as Kubernetes deployments and services, create persistent volumes (e.g., mounting /models and /request for model files and data traces), and manage load tester jobs. This ensures consistent and reproducible results across different Kubernetes or OpenShift environments.
• Comprehensive Load Testers: FMPERF provides three types of workload specifications:
• Homogeneous Workload Spec: Defines fixed input/output tokens for consistent and reproducible stress testing.
• Heterogeneous Workload Spec: Allows for random distributions of input/output tokens, ideal for testing performance under diverse request patterns.
• Realistic Workload Spec: Based on statistical models derived from real production traces, offering highly realistic workload patterns.
• LLM-Specific Metrics: Beyond standard throughput and latency, FMPERF extracts crucial LLM metrics like "time to first token," "inter-token latencies," and "prefill latencies." It also collects GPU metrics (using tools like Nvidia SMI) to derive energy efficiency metrics.
• CI/CD Integration: The Python library nature of FMPERF makes it easy to integrate into CI/CD pipelines, enabling automated performance benchmarking and regression testing for server optimizations.

Using FMPERF involves defining three Python classes:

1. Cluster Spec: Configures the Kubernetes cluster connection (local or remote).
2. Workload Spec: Defines the load tester type and its parameters.
3. Model Spec: Specifies the AI model to be benchmarked, its configurations (e.g., quantization), and resource requests (GPU, CPU, memory).

Once these are defined, a single run_benchmark function orchestrates the entire process, including server deployment, workload generation, data collection, and result summarization.

GPU Intensive Workload Benchmarking

For general GPU intensive workload benchmarking, the open-source GPU burn tool is recommended. It provides a straightforward way to stress test GPUs by consuming specified amounts of memory and running for a defined duration. This tool, combined with Nvidia SMI or GPU stats, helps assess the raw performance and stability of GPU hardware under heavy load.

Demo / Proof of Concept

▶ Watch: Benchmarking generative AI models with GI PF (6:30)

The talk included two live demonstrations, showcasing the practical application of Triton Inference Server and FMPERF.

Triton Inference Server Demo (Yuan Chen)

Yuan Chen demonstrated a basic benchmarking workflow using the Triton Inference Server.

1. Model Preparation: The demo began by showing a pre-populated model repository containing various models, including an Ernie model. Each model had its own directory, configuration, and a version1 subdirectory holding the actual model file, which was noted to be quite large.
2. Triton Server Deployment: A simple Kubernetes YAML file was used to deploy the Triton server. The YAML specified mounting the model repository, collecting a "bunch of metrics," and configuring the HTTP and gRPC ports. The server log confirmed that multiple models were successfully populated and Triton was listening on the configured ports.
3. Client-Side Benchmarking: The Performance Analyzer was then used as a containerized client. The command specified sending gRPC requests to the Triton server for the ResNet50 model, measuring the 95th percentile latency, and maintaining a constant request rate of 200 requests per second.
4. Real-time Monitoring: During the client's execution, Nvidia SMI was used to monitor the GPU utilization, which showed a steady 21-22% utilization, indicating the GPU was actively processing the inference requests.
5. Results: After a short run, the Performance Analyzer output displayed detailed metrics, including the total number of requests sent (e.g., 3,600 requests), various latency percentiles, and throughput figures, confirming the success of the basic benchmark setup.

FMPERF Demo (Chen Wang)

Chen Wang then presented a quick demo of FMPERF, highlighting its simplified, code-first approach for GenAI benchmarking.

1. Environment Setup: The demo started by verifying that local folders were mounted as persistent volumes for models (/mount/models) and request traces (/mount/request) within the Kubernetes cluster.
2. Benchmarking Script: A concise Python script for benchmarking a vLLM server was shown. The script involved defining the ClusterSpec, WorkloadSpec (using the default FMPERF load tester image), and ModelSpec classes. The run_benchmark function was then called with parameters for experiment repetitions and concurrent users.
3. Automated Kubernetes Deployment: Upon executing the Python script, kubectl get pods commands were used to observe FMPERF's automation. It first created the vLLM server deployment, which took some time to boot up. Once the server was ready, FMPERF automatically launched an FMPERF generator pod (the load tester) and an evaluator pod to send requests and collect data.
4. Concurrent Experiments: The demo showed two concurrent experiments running: one emulating a single user continuously sending requests and another with two concurrent users.
5. Result Analysis: FMPERF generated two types of results:

• A CSV summary file providing aggregated statistics across repetitions, including throughput, prefill latencies, and inter-token latencies.
• A detailed JSON file containing timestamped information for each token, decoding times, and inter-token latencies, allowing for granular analysis.

GPU Burn (Mentioned)

While not demonstrated, GPU burn was briefly mentioned as a useful, open-source, containerized tool for intense GPU workload benchmarking. It allows users to specify memory consumption and run duration to stress-test GPUs, with monitoring via Nvidia SMI or GPU stats.

Operational Implications and Optimization Strategies

▶ Watch: Monitoring GPU utilization with SMI & stats (7:00)

This talk, while not focused on security, provides crucial insights for the operational efficiency, reliability, and optimization of AI workloads in production Kubernetes environments. The methodologies and tools discussed directly contribute to making AI systems more robust, performant, and cost-effective.

1. Informed Resource Allocation and Cost Optimization: By systematically benchmarking AI models with tools like Triton's Performance Analyzer and FMPERF, operators can gain a deep understanding of actual GPU, CPU, and memory requirements. This data is vital for:

• Right-sizing deployments: Preventing over-provisioning (which wastes resources and increases cost) or under-provisioning (which leads to performance degradation).
• Optimizing GPU sharing strategies: As highlighted by previous work, benchmarking different GPU sharing techniques (time-slicing, MPS) helps identify the most efficient strategy for specific workload types, maximizing GPU utilization and ROI.
• Capacity planning: Accurate benchmarks enable better forecasting of infrastructure needs as AI model usage scales.

1. Performance Regression Detection and CI/CD Integration: FMPERF's "code-first" Python API and Kubernetes-native automation are powerful for integrating performance benchmarking into CI/CD pipelines. This allows development teams to:

• Automate performance testing: Every code change or model update can trigger a benchmark run, automatically detecting performance regressions early in the development cycle.
• Validate optimizations: Quantify the impact of server optimizations, model quantization, or new hardware on key performance metrics.
• Ensure consistent performance: Maintain a high standard of performance across different versions and environments.

1. Enhanced Reliability and Fault Recovery: The talk briefly touches upon the importance of "reliability and fault recovery" for AI workloads. While the benchmarking tools primarily focus on performance, understanding a system's performance under various loads (including extreme ones simulated by tools like GPU burn) can indirectly inform resilience strategies. By knowing how a system behaves at its limits, operators can design more robust auto-scaling policies, implement graceful degradation, and plan for recovery mechanisms.

1. Deep Understanding of Generative AI Behavior: The specialized metrics provided by GI PF and FMPERF (time to first token, inter-token latency) are critical for GenAI. These metrics directly correlate with user experience for interactive applications. Operators can use this information to:

• Tune models and serving frameworks: Optimize for responsiveness rather than just raw throughput.
• Evaluate different LLM architectures: Compare the real-world performance implications of various foundation models.
• Inform user experience design: Set realistic expectations for GenAI application latency.

1. Infrastructure Agnosticism and Reproducibility: The Kubernetes-native approach of FMPERF ensures that benchmarks are consistent and reproducible across different underlying infrastructures (cloud providers, on-premise clusters). This is crucial for:

• Vendor lock-in avoidance: Allowing organizations to evaluate and compare performance across different hardware and cloud vendors without re-architecting their benchmarking setup.
• Standardized evaluation: Providing a common framework for comparing the performance of different AI serving solutions.

In essence, the tools and methodologies presented empower AI system operators and developers to move from reactive troubleshooting to proactive optimization and strategic planning, ensuring that AI deployments are not only functional but also highly efficient, scalable, and reliable.

Key Takeaways

• Benchmarking is essential for AI workload optimization: Understanding and optimizing AI and GPU-intensive workloads in Kubernetes requires systematic benchmarking to ensure performance, scalability, and resource efficiency.
• Triton Inference Server is a versatile foundation: Nvidia Triton Inference Server, with its Performance Analyzer, offers a robust, open-source solution for general AI model serving and benchmarking across diverse models and architectures (x86, ARM, PyTorch, ONNX).
• FMPERF simplifies GenAI benchmarking in Kubernetes: The FMPERF Python library provides a powerful, code-first, and Kubernetes-native approach to automate benchmarking for large language models and generative AI, offering crucial LLM-specific metrics like "time to first token" and "inter-token latency."
• Monitoring GPU resources is critical: Tools like Nvidia SMI and GPU stats are indispensable for real-time monitoring of GPU utilization, temperature, and memory consumption, providing vital insights into hardware behavior during benchmarks.
• Reproducibility and end-to-end analysis are key: The drive towards consistent, reproducible results and comprehensive end-to-end performance analysis (beyond just inference) is crucial for advanced AI workload optimization.
• Community and open source drive innovation: The availability of open-source tools and frameworks, coupled with community collaboration (e.g., FMPERF's donation to the serving working group), is vital for advancing AI benchmarking capabilities.

About the Speaker(s)

Yuan Chen is a Senior Solutions Architect at Nvidia. He is actively involved in the Nvidia DJX cloud initiative, focusing on building Kubernetes-optimized AI infrastructure platforms specifically designed for AI and GPU workloads. His expertise lies in developing and deploying scalable AI solutions within cloud-native environments.

Chen Wang is a Senior Research Scientist at IBM Research. He is a highly active contributor to various Kubernetes Special Interest Groups (SIGs), including autoscaling, scheduling, and the serving community. His contributions extend to the working group of serving, where he plays a key role in developing and advancing benchmarking tools like FMPERF for AI inference in Kubernetes.