QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Speakers: Yujun Lin (MIT); Haotian Tang (MIT); Shang Yang (MIT); Zhekai Zhang (MIT); Guangxuan Xiao (MIT); Song Han (Professor, MIT)

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=BipXPh47ILQ

Overview

This article delves into QServe, a groundbreaking system and algorithm co-design for the efficient serving of Large Language Models (LLMs) on cloud infrastructure. Presented by Shang Yang, a second-year PhD student at MIT ECS, under the advisement of Professor Song Han, QServe introduces a novel mixed-precision quantization scheme dubbed W4A8KB4 (Quarter-Octo-Quarter). This approach quantizes model weights to 4 bits, activations to 8 bits, and the KV (Key-Value) cache, a critical component for attention mechanisms, also to 4 bits. The core motivation behind QServe is to tackle the pervasive challenge of high computational and memory demands during LLM inference, particularly in the decoding stage, which often becomes the system bottleneck in cloud serving scenarios.

The talk highlights that while prior efforts have explored aggressive quantization (e.g., W4A4), they often suffer from significant accuracy degradation and substantial inference overheads due to complex dequantization steps. QServe aims to strike an optimal balance, leveraging the memory efficiency of 4-bit weights and KV cache alongside the high-performance capabilities of NVIDIA GPU's INT8 Tensor Cores for activation computations. By meticulously co-designing the quantization algorithms and the underlying system kernels, QServe achieves substantial throughput improvements and memory savings without compromising model accuracy, thereby offering a practical and highly efficient solution for deploying large-scale LLMs.

The significance of QServe lies in its ability to unlock unprecedented efficiency for LLM serving. By carefully considering the interplay between quantization precision choices and hardware capabilities, the research delivers a system that can achieve up to 3.5 times speedup compared to state-of-the-art LLM serving frameworks like NVIDIA's TensorRT-LLM, while maintaining superior perplexity compared to other aggressive 4-bit quantization methods. This makes QServe a critical advancement for reducing the operational costs and increasing the accessibility of sophisticated LLM services, pushing the boundaries of what is achievable in resource-constrained deployment environments.

Background

▶ Watch: Introducing QServe: W4A8KB4 Quantization for LLM Serving (0:00)

The rapid proliferation of Large Language Models has brought about a paradigm shift in AI applications, yet their deployment in real-world scenarios, especially on cloud servers, presents significant engineering challenges. The sheer size of these models, often comprising billions of parameters, translates into immense computational and memory requirements, making efficient inference a primary concern. The decoding stage, where the model generates output tokens sequentially, is consistently identified as the system bottleneck. Within this stage, two operations dominate the computational cost: attention computation and General Matrix Multiplication (GEMM), which together consume the majority of decoding time.

To mitigate these costs, quantization has emerged as a crucial technique, reducing the precision of model parameters and activations to lower bit-widths (e.g., 8-bit, 4-bit) from the standard FP16 or FP32. Existing quantization schemes typically focus on different aspects:

• W4A16: Quantizes weights to 4 bits while keeping activations at FP16. This saves weight memory but often requires dequantization to FP16 before computation, introducing overhead.
• W8A8: Quantizes both weights and activations to 8 bits. This can leverage efficient INT8 Tensor Cores on modern GPUs, but 8-bit weights might still be too large for memory-bound scenarios.
• W4A4: The most aggressive approach, quantizing both weights and activations to 4 bits. While offering maximum memory compression, state-of-the-art W4A4 methods often suffer from two major drawbacks: significant accuracy degradation (as measured by perplexity) compared to higher precision models, and substantial inference overhead. This overhead stems primarily from the need for frequent dequantization operations within the main computation loop, which often execute on slower CUDA cores rather than the highly optimized Tensor Cores. These dequantization steps also increase register pressure, further impeding performance.

Furthermore, the KV cache, which stores intermediate key and value tensors from previous tokens to accelerate attention computation in autoregressive decoding, consumes a significant portion of GPU memory, especially for long context lengths. Efficiently compressing the KV cache is therefore vital for enabling longer sequences and larger batch sizes. While 4-bit KV cache quantization (KB4) offers memory savings, naive application can lead to severe accuracy loss, particularly for the key cache, which often exhibits non-uniform data distributions with prominent outliers. The challenge, therefore, is to develop a quantization strategy that not only aggressively compresses the model and KV cache but also maintains high accuracy and leverages specialized hardware efficiently, avoiding the pitfalls of prior aggressive quantization approaches. QServe directly addresses these limitations by introducing a co-designed algorithm and system infrastructure that targets the "best of both worlds" in terms of accuracy, memory efficiency, and computational throughput.

Key Findings

▶ Watch: Limitations of W4A4: Accuracy and Overhead (2:00)

QServe introduces a novel W4A8KB4 quantization and system co-design, delivering a superior trade-off between LLM serving efficiency and model accuracy. The key findings and contributions are:

• QQQ Quantization Algorithm (Quarter-Octo-Quarter): QServe's core algorithmic contribution is the QQQ quantization scheme, which sets weights to 4-bit (W4), activations to 8-bit (A8), and the KV cache to 4-bit (KB4). This precision choice is strategically designed to optimize for memory bandwidth (W4, KB4) and efficient utilization of NVIDIA's INT8 Tensor Cores (A8).
• Progressive Group Quantization for W4A8 GEMM: To achieve W4A8 GEMM without accuracy loss or dequantization overheads in the main computation loop, QServe employs a two-step progressive quantization. FP16 weights are first quantized to 8-bit (per-channel) and then further to 4-bit (per-group). Crucially, these 4-bit weights are dequantized back to 8-bit before the GEMM operation, ensuring all computations can leverage the efficient INT8 Tensor Cores. A protective range for the initial INT8 quantization step prevents overflow during the subsequent INT4-to-INT8 dequantization.
• Smooth Attention for KV Cache Compression: Recognizing the challenge of quantizing key caches with outlier channels, QServe adopts a "smooth attention" approach. Inspired by SmoothQuant, this method keeps the query matrix in FP16 (as it's relatively small) and migrates the quantization difficulty from the key cache to the query. Scaling factors are computed only for the key cache based on the absolute maximum values per channel, allowing for effective 4-bit KV cache compression with minimal accuracy degradation.
• System-Level Co-design for Performance: The QServe framework is not just an algorithm but a tightly integrated system.
• Activation Quantization Fusion: Activation quantization is fused into preceding layers (e.g., LayerNorm) whenever possible, minimizing overhead.
• Optimized W4A8 GEMM Kernel: The custom GEMM kernel ensures all computations are performed on INT8 Tensor Cores. It employs a compute-aware weight reordering strategy to address storage-compute mismatch in mixed-precision loading and an efficient subtraction after multiplication dequantization scheme for INT4-to-INT8 conversion, leveraging register-level parallelism and guaranteeing no overflow.
• Accelerated Attention Kernel: To counteract the dequantization overhead introduced by 4-bit KV cache, QServe introduces several optimizations, including efficient INT4-to-FP16 conversion, control flow simplification, and zero-point/scaling factor prefetching. These techniques collectively accelerate the attention kernel by 50% compared to TensorRT-LLM's 8-bit KV cache implementation.
• Significant Throughput Speedups: Across various LLM models and GPU types (e.g., A100), QServe achieves impressive throughput speedups, ranging from 2.4x to 3.5x over state-of-the-art serving frameworks like NVIDIA TensorRT-LLM (FP16) and leading W4A4 implementations (Atom, Quarto).
• Superior Accuracy Preservation: Compared to existing W4A4 quantization algorithms, QServe's W4A8KB4 method demonstrates significantly better perplexity preservation, indicating a superior balance between compression and model fidelity.

These findings collectively establish QServe as a state-of-the-art solution for efficient LLM serving, demonstrating that careful co-design of quantization algorithms and system implementations can yield substantial performance gains without compromising the quality of the generative models.

Technical Deep Dive

▶ Watch: QQQ Algorithm: Progressive W4A8 GEMM Quantization (3:08)

QServe's technical prowess stems from its meticulous co-design of the QQQ quantization algorithm and a highly optimized serving system. This section elaborates on the intricate details of its components.

QQQ Quantization Algorithm

The W4A8KB4 scheme is central to QServe. It involves:

• Weights (W4): Model weights are quantized to 4 bits to maximize memory bandwidth savings during loading.
• Activations (A8): Activations are quantized to 8 bits to leverage the highly efficient INT8 Tensor Cores available on modern NVIDIA GPUs.
• KV Cache (KB4): The Key-Value cache for attention is quantized to 4 bits per element, significantly reducing memory footprint and enabling longer context windows and larger batch sizes.

Progressive Group Quantization for W4A8 GEMM

For the core W4A8 GEMM operations, QServe employs a sophisticated two-level progressive group quantization:

1. FP16 to INT8: The original FP16 weights are first quantized to 8-bit integers using per-channel quantization. This step ensures a fine-grained representation.
2. INT8 to INT4: These intermediate 8-bit weights are then further quantized to 4-bit integers using per-group quantization. This balances compression with the ability to capture weight distributions effectively.

A critical design choice is to dequantize the 4-bit weights back to 8-bit before they are fed into the Tensor Cores. This allows the computation to fully utilize the high throughput of INT8 Tensor Cores. To prevent overflow during this INT4-to-INT8 dequantization, QServe introduces a protective range for the initial FP16-to-INT8 quantization step. Specifically, the intermediate 8-bit weight values are restricted to a range (e.g., -119 to 119). This mathematical constraint guarantees that when a 4-bit value is dequantized to 8 bits, it will always fit within the INT8 range, preventing numerical instability and ensuring accuracy. The per-channel scaling factors are applied in the epilog stage of the GEMM kernel to minimize overhead in the main computation loop.

Smooth Attention for K4V4 KV Cache Compression

Directly quantizing the KV cache to 4 bits is challenging, particularly for the key cache, which often exhibits significant outliers in its data distribution. The value cache, conversely, tends to have a more normalized distribution and is easier to quantize. QServe addresses this with Smooth Attention, drawing inspiration from the SmoothQuant technique.

• The query matrix is kept in FP16. This is a strategic choice because the query matrix is typically much smaller than the KV cache, making its FP16 memory footprint negligible.
• By keeping the query in FP16, the "quantization difficulty" (i.e., the need to handle outliers) is effectively migrated from the key cache to the query matrix.
• Scaling factors ($\lambda_I$) for the key cache are computed on a per-channel basis, solely considering the absolute maximum value of the key cache in each channel. The talk specifically mentions using an $\alpha$ value of 0.5 in the scaling factor computation to achieve a balance between algorithm complexity and accuracy. This allows the key cache to be effectively compressed to 4 bits without severe accuracy degradation.

QServe System Design

The QServe system is engineered to maximize the benefits of the W4A8KB4 quantization scheme.

Precision Data Flow and Activation Quantization Fusion

In QServe's data flow, every linear layer in the decoder operates with 4-bit weights (W4) and 8-bit input activations (A8). To minimize the overhead associated with activation quantization, QServe employs quantization fusion. Whenever possible, the quantization of activations is fused into the preceding layers, such as LayerNorm. This means the output of LayerNorm is directly quantized to 8 bits, eliminating explicit quantization kernels and reducing memory transfers.

Quantized Matrix Multiplication (GEMM) Kernel on GPUs

State-of-the-art GEMM implementations for FP16 and W8A8 typically execute the main computation loop entirely on Tensor Cores, offloading dequantization and scaling to the epilog for efficiency. However, for mixed-precision operations like W4A16 or W4A4, dequantization often occurs within the main loop on slower CUDA cores, leading to performance bottlenecks and increased register pressure.

QServe's W4A8 GEMM kernel is designed to avoid these pitfalls. By dequantizing 4-bit weights to 8 bits before the Tensor Core computation, it ensures that the entire GEMM main loop runs on INT8 Tensor Cores. The per-channel scaling operations are moved to the epilog, similar to FP16/W8A8 efficient implementations, maintaining high throughput. The choice of weight dequantization (INT4 to INT8) over partial sum dequantization is crucial as it incurs lower register pressure.

Challenges and Design Details of QServe GEMM Kernel

1. Compute-Aware Weight Reordering:

• Problem: Standard load matrix instructions on GPUs are optimized for uniform data access. For mixed-precision (e.g., loading 4-bit weights for 8-bit computation), a single thread might load elements intended for multiple threads in a warp, leading to a storage-compute mismatch and incorrect data distribution for Tensor Core operations.
• Solution: QServe introduces compute-aware weight reordering. 4-bit weights required by a specific thread are packed together into 8-bit blocks before loading. This reordering allows the load matrix instruction to efficiently fetch the correct 8-bit elements for each thread, resolving the mismatch and enabling seamless Tensor Core utilization.

1. Efficient INT4 to INT8 Dequantization:

• Weight Unpacking: QServe efficiently unpacks 4-bit weights into 8-bit representations. For example, 32 4-bit weights can be unpacked using only three specialized instructions, demonstrating high efficiency.
• Applying Zero Points and Scaling Factors: After unpacking, zero points and scaling factors must be applied to convert the raw INT4 values to their dequantized INT8 representations.
• "Subtraction before multiplication": An intuitive approach (add zero points, then multiply by scale) carries a risk of overflow, potentially leading to incorrect results.
• "Subtraction after multiplication": QServe adopts this safer approach. The progressive group quantization algorithm guarantees that this method will not lead to overflow. Furthermore, this design allows for register-level parallelism, where multiple 4-bit weight elements can be dequantized simultaneously with a single instruction, significantly accelerating the process.

Attention Computation Acceleration

While 4-bit KV cache reduces memory, a naive implementation can make the attention kernel slower due to the overhead of dequantization operations, shifting the bottleneck from memory to computation. QServe addresses this by:

• Efficient INT4 to FP16 Conversion: Optimized routines for converting 4-bit KV cache elements to FP16 for the attention score calculation.
• Control Flow Simplification: Streamlining the kernel's control logic to reduce instruction cycles.
• Zero-Point and Scaling Factor Prefetching: Strategically prefetching these values to ensure they are available when needed, minimizing memory latency.

These optimizations collectively accelerate the attention kernel by approximately 50% compared to the baseline 8-bit KV cache implementation in TensorRT-LLM, ensuring that the memory benefits of KB4 translate into real-world performance gains.

Experimental Setup & Results

▶ Watch: QServe System Overview: Precision Data Flow (6:54)

The QServe system was rigorously evaluated against state-of-the-art LLM serving frameworks to demonstrate its performance and accuracy advantages.

Baselines and Comparison Targets:

• NVIDIA TensorRT-LLM (FP16): Representing the high-performance baseline for standard FP16 precision serving.
• Atom and Quarto: Leading implementations for W4A4 quantization, which serve as direct competitors in the aggressive low-bit quantization space.

Hardware and Workloads:

Experiments were conducted on NVIDIA A100 GPUs, a standard for cloud LLM serving. The evaluation spanned different LLM models (specific models were not explicitly named in the transcript, but implied to be large-scale generative models) and various input batch sizes, including super large batches of 128 and 256, pushing the limits of GPU memory and computational capacity. The focus was primarily on the decoding stage, which is the performance bottleneck in LLM serving.

Metrics:

• Throughput (Speedup): The primary performance metric, measuring the number of tokens processed per second, relative to baselines.
• Perplexity: The standard metric for evaluating the accuracy and quality of language models, indicating how well the model predicts a sample of text. Lower perplexity indicates better accuracy.

Headline Results:

• Throughput Speedup: QServe achieved remarkable speedups, demonstrating 2.4x to 3.5x higher throughput compared to the state-of-the-art frameworks (TensorRT-LLM, Atom, Quarto) across the tested models and GPU configurations. This significant improvement underscores QServe's efficiency in processing LLM inference requests.
• Accuracy Preservation: In terms of perplexity, QServe's W4A8KB4 quantization algorithm significantly better preserved model perplexity compared to state-of-the-art W4A4 quantization algorithms. This indicates that QServe successfully balances aggressive quantization for efficiency with maintaining the generative quality of the LLMs, avoiding the severe accuracy degradation often seen with other highly compressed models.

Detailed Performance Breakdown (as discussed in Q&A):

• The speaker acknowledged that dequantization operations, even when optimized, still incur some overhead. However, QServe's design minimizes this overhead by keeping it outside the Tensor Core's main computation loop and leveraging register-level parallelism.
• For longer context models, the overhead of GEMM dequantization becomes less significant as the attention operation starts to dominate the overall computation time. However, very long sequences introduce new challenges for the attention kernel, particularly concerning the size of prefetchable scaling factors and zero points. This specific challenge is being addressed in a follow-up project, Elsa.
• For larger models (i.e., more layers), the relative overhead of GEMM and attention computations remains roughly the same, meaning QServe's benefits scale proportionally without introducing new bottlenecks specific to model depth.
• The system is specifically designed to handle large batch sizes (128, 256), optimizing for scenarios where GPU memory is fully utilized to maximize throughput.

In summary, QServe's experimental results unequivocally demonstrate its superior performance-accuracy trade-off, solidifying its position as a leading solution for efficient and high-quality LLM serving.

Practical Implications

▶ Watch: Quantized GEMM Implementation on GPUs (8:04)

QServe's innovations have profound practical implications for various stakeholders involved in deploying and managing Large Language Models in production environments.

For practitioners and model deployers, QServe offers a significant leap in efficiency. The achieved 2.4x to 3.5x throughput speedup directly translates into higher query-per-second (QPS) capacity for a given GPU infrastructure or, conversely, a substantial reduction in the number of GPUs required to serve a specific workload. This leads to considerable cost savings on cloud computing resources, making LLM services more economically viable for a broader range of applications and user bases. The superior perplexity preservation means that these efficiency gains do not come at the cost of model quality, ensuring that the deployed LLMs continue to deliver accurate and coherent outputs. This enables practitioners to deploy larger, more capable models in more constrained environments or serve more users with existing hardware.

For infrastructure teams and MLOps engineers, QServe provides a robust blueprint for building highly optimized LLM serving systems. It highlights the critical importance of co-design—not just optimizing the model or the hardware in isolation, but meticulously tailoring quantization algorithms to leverage specific hardware features (like INT8 Tensor Cores) and crafting custom kernels that minimize overheads. The techniques demonstrated, such as compute-aware weight reordering, fused activation quantization, and optimized dequantization schemes, offer valuable insights and strategies for developing efficient inference engines for mixed-precision models. The ability to handle super large batch sizes (e.g., 128, 256) is also crucial for maximizing GPU utilization and throughput in data center environments.

For model builders and researchers, QServe presents a compelling case for W4A8KB4 as an optimal mixed-precision choice for LLM serving. It demonstrates that aggressive 4-bit weight and KV cache quantization, when combined with 8-bit activations and careful algorithmic design (like progressive group quantization and smooth attention), can outperform purely 4-bit (W4A4) approaches in terms of both accuracy and effective throughput. This informs future research directions in quantization-aware training and post-training quantization, guiding the development of models that are inherently more amenable to efficient deployment.

Tradeoffs and Limitations: While QServe delivers significant advancements, it's important to acknowledge certain tradeoffs and limitations:

• Dequantization Overhead: Although minimized through careful kernel design (e.g., moving scaling to epilog, register-level parallelism), dequantization operations still exist. This means there is always a slight computational cost compared to a hypothetical scenario where no dequantization is needed, though QServe makes this cost negligible by executing it on fast Tensor Cores or in parallel.
• Long Context Challenges: For extremely long sequences, the attention kernel can face new challenges. Specifically, the size of scaling factors and zero points for the KV cache might become too large to be effectively prefetched, potentially impacting performance. The speakers explicitly mentioned that this is an active area of research for their group, addressed in a separate project named Elsa. This implies that while QServe greatly improves efficiency for typical long contexts, ultra-long contexts may require further specialized optimizations.
• Hardware Specificity: The system design heavily leverages NVIDIA's INT8 Tensor Cores. While this provides excellent performance on NVIDIA GPUs, adapting QServe to other hardware platforms (e.g., AMD GPUs, custom AI accelerators) would require re-engineering the custom kernels to suit their specific architectural features and instruction sets.

Despite these considerations, QServe represents a robust, high-performance, and economically attractive solution for deploying cutting-edge LLMs in demanding cloud environments, offering a clear path towards more sustainable and scalable AI services.

Key Takeaways

• W4A8KB4 (QQQ) as an Optimal Precision Choice: QServe demonstrates that a mixed-precision scheme of 4-bit weights, 8-bit activations, and 4-bit KV cache offers a superior balance for LLM serving, optimizing memory bandwidth, INT8 Tensor Core utilization, and KV cache efficiency.
• Algorithm-System Co-Design is Crucial: Realizing the full benefits of quantization requires a tight integration of algorithmic innovations (e.g., progressive group quantization, smooth attention) with meticulously optimized system kernels (e.g., compute-aware weight reordering, fused operations).
• Progressive Quantization Prevents Overflow and Enhances Accuracy: The two-step progressive group quantization for weights, with a protective INT8 range and "subtraction after multiplication" dequantization, effectively prevents overflow and preserves model perplexity during W4A8 GEMM.
• Smooth Attention Enables Efficient KV Cache Compression: By keeping queries in FP16 and migrating quantization difficulty, Smooth Attention allows for effective 4-bit KV cache compression, significantly reducing memory footprint without compromising attention accuracy.
• Significant Throughput Gains: QServe achieves substantial throughput speedups (2.4x to 3.5x) over state-of-the-art LLM serving frameworks, offering considerable cost savings and higher inference capacity on A100 GPUs.
• Practical for Cloud LLM Deployment: The research provides a practical, high-performance solution for efficient LLM serving on cloud infrastructure, enabling larger models, longer contexts, and higher batch sizes with minimal accuracy degradation.

About the Speaker(s)

The talk was primarily presented by Shang Yang, a second-year PhD student in Electrical Engineering and Computer Science (EECS) at MIT. He is advised by Professor Song Han, a renowned faculty member at MIT, whose research group focuses on efficient AI computing. The work presented is a collaborative effort from Professor Han's group, with additional contributors including Yujun Lin, Haotian Tang, Zhekai Zhang, and Guangxuan Xiao, all affiliated with MIT. Their collective expertise in deep learning, hardware-aware AI, and system optimization underpins the innovative co-design approach of QServe.