ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

Speakers: Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Ana Klimovic, Eiko Yoneki

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The proliferation of large language models (LLMs) has revolutionized many industries, yet their deployment in production environments presents significant challenges, primarily due to the immense computational resources they demand. ThunderServe, a novel system presented at MLSys 2025 by Taiyi Wang from the University of Cambridge, in collaboration with Peking University and ETH, addresses the critical need for high-performance and cost-efficient LLM serving, particularly within dynamic cloud environments. The core motivation behind ThunderServe is to tackle the prohibitive expenses associated with LLM inference, which often require substantial GPU resources to meet stringent latency and throughput requirements, while simultaneously combating the pervasive issue of underutilized hardware, especially older-generation GPUs in cloud data centers.

This talk introduces a paradigm shift in LLM serving by proposing a wise phase splitting approach, which differentiates between the compute-bound prefill phase (processing input tokens) and the memory bandwidth-bound decoding phase (generating output tokens). ThunderServe intelligently maps these distinct phases to heterogeneous GPU resources available in the cloud, leveraging the specific strengths of different GPU types. For instance, NVIDIA A40 GPUs, often cheaper, are identified as more suitable for the prefill phase, while NVIDIA GeForce RTX 3090 Ti GPUs are better suited for the decoding phase due to their memory bandwidth characteristics. The system's innovative heterogeneity-aware scheduler, coupled with efficient communication optimizations like KV cache compression and a lightweight rescheduling mechanism, enables it to adapt to fluctuating workloads and resource availability without incurring costly model reloads.

ThunderServe is designed to maximize end-to-end throughput and SLA (Service Level Agreement) attainment on cloud infrastructures at minimal cost. By strategically utilizing diverse and often cheaper cloud GPUs, the system not only reduces operational expenditures but also significantly enhances performance. The research demonstrates that ThunderServe can achieve up to 25x lower latency deadlines for a given SLA target compared to state-of-the-art baselines, marking a substantial leap forward in the practical deployment of LLMs. This work is crucial for any organization looking to deploy LLMs at scale in the cloud, offering a blueprint for optimizing resource utilization, reducing costs, and improving service quality in the face of complex and dynamic infrastructure challenges.

Background

▶ Watch: Introduction to ThunderServe: high-performance, cost-efficient LLM serving. (0:00)

The rapid advancement and widespread adoption of LLMs have made them central to many AI applications, from content generation to intelligent chatbots. However, deploying these models in production, especially for real-time inference, remains a formidable challenge. LLMs are notorious for their massive parameter counts, often ranging from billions to trillions, which translate into immense computational demands during inference. Meeting the latency and throughput requirements for online services typically necessitates significant GPU resources, leading to high operational costs. This economic burden is exacerbated by the fact that cloud environments, while offering elasticity, also present a complex landscape of diverse and often underutilized hardware.

A key observation underpinning ThunderServe's approach is the distinct computational characteristics of the two primary phases of LLM inference:

1. Prefill Phase (Context Processing): This phase processes the input prompt, computing the initial key-value (KV) cache for all input tokens. It is typically compute-bound, requiring high floating-point operations per second (FLOPS) to perform matrix multiplications for the attention mechanism over the entire input sequence.
2. Decoding Phase (Token Generation): This phase iteratively generates one token at a time, using the previously computed KV cache and the newly generated token to predict the next. This phase is predominantly memory bandwidth-bound, as it involves frequent memory accesses to the KV cache and model weights for each token generation step.

This inherent phase splitting presents a significant opportunity for optimization. Different types of GPUs exhibit varying strengths; some are more compute-intensive, while others offer higher memory bandwidth. For instance, the talk highlights that GPUs like the NVIDIA A40, often found at a lower cost in cloud environments, are more efficient for compute-bound prefill tasks, whereas NVIDIA GeForce RTX 3090 Ti GPUs with their higher memory bandwidth are better suited for memory-bound decoding. The challenge, then, is to intelligently match these heterogeneous GPU strengths to the specific demands of each LLM inference phase.

However, leveraging this opportunity in cloud environments introduces several complexities:

• Heterogeneous and Limited Resource Pool: Cloud providers offer a diverse array of GPU types (e.g., A100, V100, A40, 3090 Ti), often from different hardware generations or even different vendors. The availability of specific types is dynamic and can be restricted, making static resource allocation suboptimal.
• Heterogeneous and Potentially Low Network Bandwidth: GPUs within cloud clusters are typically interconnected via network fabrics (e.g., Ethernet), which can exhibit varying bandwidths. While the speaker noted that "interconnection bandwidth is not the bottleneck" in some contexts, the initial design consideration acknowledges that "low network bandwidth also limits your design," particularly when transferring large data structures like the KV cache between phases running on different GPUs.
• Workload Variability: The nature of online LLM services means that both the availability of cloud resources and the distribution of incoming requests can change over time. GPUs might become occupied, and the mix of long vs. short input sequences or generation lengths can shift, requiring an adaptive serving system that can adjust without significant downtime or performance degradation.

Prior work in LLM serving, such as HexGen, DistServe, and vLLM, has focused on various optimizations like continuous batching, KV cache management, and different parallelism strategies. However, these systems often assume a homogeneous GPU environment or do not fully exploit the nuanced characteristics of heterogeneous cloud resources and dynamic workloads. ThunderServe aims to bridge this gap by offering a comprehensive solution that explicitly addresses these cloud-specific challenges.

Key Findings

▶ Watch: Challenges of LLM serving in cloud environments. (2:00)

ThunderServe introduces a sophisticated approach to LLM serving that significantly enhances performance and cost-efficiency in heterogeneous cloud environments. The key findings and contributions of this work are centered around its intelligent resource management, adaptive scheduling, and efficient communication strategies:

• Wise Phase Splitting with Heterogeneity Awareness: The system's foundational discovery is the optimal execution of the prefill and decode stages on distinct groups of heterogeneous GPUs. By carefully matching the compute-bound nature of prefill with GPUs excelling in raw computational power (e.g., A40) and the memory bandwidth-bound nature of decode with GPUs offering high memory bandwidth (e.g., 3090 Ti), ThunderServe achieves a more efficient utilization of diverse hardware. This approach inherently reduces cost by allowing the use of cheaper, older-generation GPUs for specific tasks where they are most effective.

• Strong Heterogeneity-Aware Scheduler: At the heart of ThunderServe is an advanced scheduler that dynamically determines how to partition available GPUs into different groups and assign them to either the prefill or decode phases. This scheduler also defines the optimal parallelism configurations within each group, considering both intra- and inter-node communication costs. Its two-level optimization strategy, combining high-level GPU grouping with low-level parallelization, is crucial for adapting to the complex cloud landscape.

• Efficient Communication Optimizations via KV Cache Compression: Recognizing the potential for network bottlenecks when transferring data between prefill and decode phases, especially the large KV cache, ThunderServe implements an innovative KV cache compression technique. By quantizing and packing KV cache values before transmission and dequantizing/unpacking upon reception, the system significantly mitigates communication overhead. This optimization ensures that the communication cost remains a minimal fraction of the end-to-end processing time, preserving performance even with lower network bandwidths.

• Lightweight Rescheduling for Online Adaptability: Cloud environments and online workloads are inherently dynamic. ThunderServe addresses this with a lightweight rescheduling mechanism that can adjust to shifts in workload patterns (e.g., changing input sequence lengths, request distributions) and node availability without requiring expensive model reloads or complete rebuilding of GPU groups. This adaptability is achieved by primarily modifying phase designations and orchestration strategies, making the system robust and responsive in real-time.

• Maximized Throughput and SLA Attainment at Minimum Cost: The overarching goal and a core finding of ThunderServe is its ability to maximize end-to-end throughput and meet strict SLA targets on cloud infrastructures while minimizing cost. By intelligently integrating all these features, ThunderServe demonstrates superior performance.

• Headline Experimental Results: The system's effectiveness is validated through extensive experiments on both heterogeneous cloud clusters and homogeneous in-house clusters. ThunderServe achieves up to 25x lower latency deadlines for a given SLA target compared to state-of-the-art baselines like HexGen, DistServe, and vLLM across various workloads (coding, conversation). Furthermore, it consistently outperforms these SOTA serving systems within the same price budget, confirming its cost-efficiency. The system is also designed to maximize GPU utilization, ensuring that resources are pushed to their capacity limits.

These findings collectively demonstrate that ThunderServe provides a comprehensive and highly effective solution for deploying LLMs in challenging cloud environments, offering significant improvements in both performance and cost-efficiency.

Technical Deep Dive

▶ Watch: ThunderServe's core features and optimizations for LLM serving. (3:20)

ThunderServe's architecture is meticulously designed to address the complexities of LLM serving in heterogeneous cloud environments. It comprises three primary components: the Scheduler, the Workload Profiler, and the Task Coordinator.

The Workload Profiler continuously monitors real-time request patterns, including input/output sequence lengths, arrival rates, and other relevant metrics. This profiler is dynamic and lightweight, ensuring that the system can adapt to evolving workload characteristics without incurring significant overhead. The information gathered by the profiler is fed to the scheduler, enabling informed decision-making.

The Scheduler is the brain of ThunderServe, responsible for generating an optimal deployment plan. This plan dictates how available GPUs are grouped, how these groups are assigned to the prefill and decode phases, and how parallelism configurations are defined within each group. The scheduler operates on a two-level optimization framework:

1. Upper-Level Optimization (GPU Grouping and Partitioning): This level focuses on deciding how to group the heterogeneous GPUs.

• It starts with a heuristics-based initialization.
• A hierarchical clustering method is employed to cluster GPUs based on their interconnection bandwidth matrix, ensuring that GPUs with good connectivity are grouped together.
• To explore the solution space efficiently, the scheduler uses Tabu search, a metaheuristic optimization algorithm. Tabu search starts from an initial solution and iteratively moves to a neighboring solution, utilizing a "tabu list" to avoid recently visited solutions and prevent cycles. The neighborhood searching involves basic operations like flip (changing a GPU's assignment), split (dividing a group), merge (combining groups), and move (relocating a GPU). This approach is described as "cheap and easy," making it suitable for dynamic adjustments.

1. Lower-Level Optimization (Parallelization Strategy and Orchestration): Once GPU groups are defined, this level determines the best parallelization strategy for each model replica and orchestrates the flow of requests between prefill and decode replicas.

• Parallelization Strategy: ThunderServe incorporates expert knowledge into its parallelization design. It explicitly disallows tensor model parallelism across GPUs located in different physical nodes to avoid high inter-node communication costs. Instead, it supports non-uniform pipeline layer partitioning, where model layers are distributed across GPUs based on their memory capacities and computational demands. A dynamic programming algorithm is utilized to optimize this pipeline model parallelism, ensuring efficient distribution of the model across the assigned GPUs.
• Orchestration: The system needs to efficiently route requests between the prefill and decode model replicas. This orchestration problem is formulated as a Two-Stage Transportation Problem (TSTP). The TSTP is then solved using linear programming, which determines the optimal flow of requests to maximize throughput and minimize latency, considering the capacities and processing capabilities of the prefill and decode groups.

The Task Coordinator receives the optimal deployment plan from the scheduler and the real-time request patterns from the profiler. Its role is to dispatch incoming LLM inference requests to the appropriate prefill and decode GPU groups, ensuring that the determined orchestration strategy is followed.

Beyond the core scheduling and orchestration, ThunderServe integrates several crucial supporting mechanisms:

• Lightweight Rescheduling: To handle online workload shifts and node changes without the prohibitive cost of rebuilding GPU groups or unloading/reloading large LLMs, ThunderServe implements a lightweight rescheduling mechanism. Instead of a full system reconfiguration, it primarily modifies the phase designations (i.e., which GPUs are assigned to prefill vs. decode) and the orchestration strategy (how requests are routed). A key insight is that the ratio of resources allocated to prefill versus decode is a critical factor influencing overall performance. By adjusting this ratio and the TSTP solution, the system can adapt efficiently, demonstrating a well-deserved tradeoff between full rebuilding and minimal disruption.

• KV Cache Compression: Communication between prefill and decode replicas, especially the transfer of the large KV cache, can become a bottleneck, particularly with potentially low network bandwidths in cloud environments. ThunderServe addresses this with KV cache compression. During the transfer from prefill to decode replicas, the KV cache values are quantized (reduced precision) and packed (compressed). Upon reception by the decode replicas, these values are immediately unpacked and dequantized. This process significantly reduces the amount of data transmitted over the network. The talk highlights that, even with compression, the performance remains high, and the communication cost becomes only a small part of the total end-to-end processing time, demonstrating the effectiveness of this optimization.

• Batching Strategy: In the Q&A, it was clarified that ThunderServe employs different batching strategies for each phase. For the prefill phase, the system typically does not batch requests to maximize latency, prioritizing the quick processing of individual prompts. In contrast, for the decode phase, strong batching is employed to optimize for throughput, which is crucial given that decoding is memory-bound and benefits significantly from parallel processing of multiple ongoing generation sequences. This nuanced batching strategy further optimizes resource utilization for the specific demands of each phase.

In summary, ThunderServe's technical depth lies in its multi-faceted approach: a sophisticated scheduler leveraging advanced optimization techniques, intelligent phase-specific resource allocation, efficient inter-phase communication, and adaptive online rescheduling, all designed to operate seamlessly within the complex and dynamic landscape of cloud-based LLM serving.

Experimental Setup & Results

▶ Watch: Detailed two-level scheduling algorithm for GPU grouping and parallelism. (5:20)

To rigorously evaluate ThunderServe's performance and cost-efficiency, comprehensive experiments were conducted across various environments and workloads, comparing it against leading LLM serving systems.

Experimental Environments:

The evaluation was performed in two distinct settings to demonstrate the system's robustness:

1. Heterogeneous Cloud Clusters: This setup mimicked real-world cloud deployments, featuring a mix of different GPU types (e.g., NVIDIA A40, 3090 Ti) with varying compute capabilities, memory bandwidths, and price points. This environment was crucial for validating ThunderServe's core hypothesis of leveraging GPU heterogeneity.
2. Homogeneous In-house Clusters: To provide a baseline and ensure the system's efficacy even in uniform environments, tests were also run on in-house clusters composed of identical GPUs. This helped isolate the benefits derived specifically from heterogeneity management.

Workloads:

The system was tested with diverse LLM workloads to reflect real-world usage patterns:

• Coding: Representing tasks with potentially longer input contexts and structured outputs.
• Conversation: Mimicking interactive chat applications, often characterized by shorter, iterative turns.

These varied workloads ensured that ThunderServe's adaptive mechanisms could handle different distributions of input and output sequence lengths, which dynamically influence the demands on the prefill and decode phases.

Baselines:

ThunderServe's performance was benchmarked against several state-of-the-art LLM serving engines widely recognized in the industry:

• HexGen: A system known for its efficiency in LLM inference.
• DistServe: Another distributed LLM serving framework.
• vLLM: A popular and highly optimized serving system that focuses on continuous batching and PagedAttention for efficient KV cache management.

These baselines represent competitive solutions in the LLM serving landscape, providing a strong comparative context for ThunderServe's innovations.

Metrics:

The primary performance metrics used for evaluation were:

• Latency: Specifically, the end-to-end latency of processing requests, crucial for interactive applications.
• SLA Attainment: The percentage of requests that meet predefined latency targets or deadlines, indicating the system's reliability in production.

The goal was to maximize throughput and SLA attainment while minimizing cost.

Headline Results:

The experimental results unequivocally demonstrated ThunderServe's superior performance and cost-efficiency:

• Significant Latency Reduction: ThunderServe exhibited remarkably low latency, achieving up to 25x lower latency deadlines for a given SLA target compared to the baselines across various experimental settings. This drastic reduction in latency is a critical improvement for real-time LLM applications.
• Outperformance within Price Budget: The system consistently outperformed other SOTA serving systems while operating within the same price budget. This highlights ThunderServe's ability to extract more performance per dollar, validating its cost-efficient design principles.
• KV Cache Compression Effectiveness: The KV cache compression technique proved highly effective. Even with compression, the system maintained high performance, and the communication cost associated with transferring KV cache data between phases was reduced to a "small part of the end-to-end processing time." This confirms that network bottlenecks, a common concern in distributed systems, were successfully mitigated.
• Maximized GPU Utilization: While specific utilization percentages were not detailed in the main talk, the Q&A session clarified that ThunderServe aims to "maximize the GPU percentages," suggesting that the system drives GPUs to their capacity limits, thus ensuring efficient resource utilization.
• Adaptive Rescheduling Benefits: The lightweight rescheduling mechanism was shown to effectively handle workload shifts and node changes without the need for expensive model reloads or full system reconfigurations, demonstrating its practical value in dynamic cloud environments. The tradeoff between lightweight and full rebuilding was validated as "well-deserved."

In conclusion, the experimental results firmly establish ThunderServe as a leading solution for high-performance and cost-efficient LLM serving, particularly in heterogeneous cloud environments. Its ability to drastically reduce latency, optimize resource utilization, and adapt to dynamic conditions sets a new benchmark in the field.

Practical Implications

▶ Watch: Lightweight rescheduling to adapt to online workload changes. (7:00)

ThunderServe's innovative approach to LLM serving carries significant practical implications for practitioners, infrastructure teams, model builders, and deployers navigating the complex landscape of large-scale AI deployments.

For Infrastructure Teams and Cloud Providers:

• Cost Efficiency: The most immediate benefit is the potential for substantial cost reduction in LLM inference. By intelligently leveraging heterogeneous and often cheaper cloud GPUs (e.g., older generations like A40s) for specific phases (like prefill), organizations can significantly lower their operational expenditure compared to exclusively relying on expensive, top-tier GPUs. This opens up opportunities for more widespread and budget-friendly LLM deployment.
• Enhanced Resource Utilization: ThunderServe maximizes the utilization of diverse GPU assets. Instead of having idle or underutilized older GPUs, the system can integrate them into the serving pipeline where their specific strengths (e.g., compute for prefill) can be optimally exploited. This leads to a more efficient use of existing hardware inventory.
• Robustness to Cloud Dynamics: Cloud environments are inherently dynamic, with fluctuating resource availability and network conditions. ThunderServe's lightweight rescheduling and KV cache compression mechanisms provide a robust solution that can adapt to these changes without costly downtime or performance degradation. This translates to more stable and reliable LLM services.

For Model Builders and Deployers:

• Improved Performance and SLA Attainment: Achieving up to 25x lower latency for a given SLA target means that LLM-powered applications can become significantly more responsive, enhancing user experience in real-time scenarios like chatbots, code assistants, or interactive content generation. This allows deployers to meet stricter performance requirements for critical applications.
• Simplified Deployment on Heterogeneous Hardware: Deploying LLMs across diverse GPU types typically involves complex manual configuration and optimization. ThunderServe's automated, heterogeneity-aware scheduler simplifies this process, abstracting away much of the underlying hardware complexity and enabling easier deployment across varied cloud instances.
• Scalability and Adaptability: The system's ability to dynamically adjust to workload shifts (e.g., varying input lengths, request spikes) ensures that LLM services can scale effectively and maintain performance under fluctuating demand, a common challenge in online services.

Tradeoffs and Limitations:

While ThunderServe offers compelling advantages, it's important to consider its tradeoffs and current limitations:

• Complexity of the Scheduler: Although the Tabu search for GPU grouping is described as a "cheap solution," the overall two-level optimization process, involving hierarchical clustering, dynamic programming for pipeline parallelism, and linear programming for orchestration, introduces a layer of algorithmic complexity. While designed for efficiency, the computational overhead of running and adapting this scheduler might be a factor for extremely rapid, sub-second reconfigurations.
• Profiling Dependency: The system relies on a Workload Profiler to inform the scheduler about real-time request patterns. The accuracy and responsiveness of this profiler are crucial for the scheduler's effectiveness. Any lag or inaccuracy in profiling could impact the optimality of the deployment plan.
• Hardware Agnosticism (Current Scope): Currently, ThunderServe has been tested primarily with NVIDIA GPUs. While the underlying principles of phase splitting and heterogeneity awareness are general, extending full support to other hardware providers like AMD GPUs or TPUs would require additional integration and optimization, as acknowledged by the speaker, due to differing hardware architectures and interconnection mechanisms. The goal is to support diverse hardware, but this remains a future development.
• Model Specificity: The optimizations are geared towards transformer-based LLMs. While widely applicable to current LLMs, novel model architectures might require re-evaluation of the prefill/decode characteristics.

In conclusion, ThunderServe provides a powerful and practical framework for deploying LLMs efficiently and cost-effectively in the cloud. Its benefits in cost reduction, performance enhancement, and adaptability make it a critical tool for organizations looking to operationalize LLM technology at scale, despite some inherent complexities and current hardware scope limitations.

Key Takeaways

• LLM Inference Phase Splitting: LLM inference can be effectively split into a compute-bound prefill phase (input processing) and a memory bandwidth-bound decoding phase (token generation), each with distinct hardware requirements.
• Leveraging Heterogeneous Cloud GPUs: ThunderServe exploits the strengths of diverse cloud GPUs by assigning them to specific phases (e.g., cheaper NVIDIA A40 for prefill, NVIDIA 3090 Ti for decode), enabling significant cost reductions and improved resource utilization.
• Advanced Two-Level Scheduling: The system features a sophisticated scheduler that uses Tabu search for high-level GPU grouping, hierarchical clustering for connectivity, and dynamic programming combined with linear programming (TSTP) for low-level parallelization and request orchestration.
• Adaptive and Efficient Mechanisms: Lightweight rescheduling allows ThunderServe to adapt to dynamic workload shifts and node changes without expensive model reloads, while KV cache compression mitigates network bottlenecks by quantizing and packing data for inter-phase communication.
• Superior Performance and Cost-Efficiency: Experimental results demonstrate that ThunderServe achieves up to 25x lower latency for a given SLA target and consistently outperforms state-of-the-art serving systems (like HexGen, DistServe, vLLM) within the same price budget on heterogeneous cloud clusters.
• Maximized Resource Utilization: The system is designed to maximize GPU utilization, ensuring that available hardware is pushed to its capacity limits for optimal throughput and efficiency in cloud environments.

About the Speaker(s)

The talk on ThunderServe was primarily presented by Taiyi Wang, a fourth-year PhD student from the University of Cambridge. This research is a collaborative effort involving multiple institutions, including Peking University and ETH Zurich, underscoring a strong academic partnership in addressing complex challenges in AI/ML systems. The full list of speakers and contributors includes Youhe Jiang, Fangcheng Fu, Xiaozhe Yao, Taiyi Wang, Ana Klimovic, and Eiko Yoneki, representing a diverse team of researchers.