Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

Speakers: Gao Wei, Xinyu Zhou, Peng Sun, Tianwei Zhang, Yonggang Wen

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

This talk, presented by Hanyu from Alibaba on behalf of the authors from Nanyang Technological University, S Lab, and Shanghai AI Lab, delves into a critical challenge in serving large language models (LLMs): the immense memory footprint of the Key-Value (KV) cache. As LLMs continue to scale in size and complexity, efficient and cost-effective serving becomes paramount. The KV cache, which stores intermediate key and value tensors for the attention mechanism, consumes a disproportionately large amount of GPU memory, often exceeding the model weights themselves. For instance, serving a Llama 3 70B model in FP16 with a batch size of 512 and a prompt length of 2048 tokens requires 130 GB for model weights but a staggering 512 GB for the KV cache alone.

The core of this work is a comprehensive re-evaluation of existing KV cache compression techniques, including both quantization and sparsity-based methods. While a plethora of research has proposed various algorithms, their real-world efficacy and suitability for production environments remain largely unverified. This paper addresses this gap by conducting an extensive literature review of over 40 papers and performing a rigorous quantitative analysis of selected state-of-the-art compression techniques. The authors identify key limitations and underexplored aspects of these methods, particularly concerning their interaction with modern LLM serving frameworks and their impact on end-to-end performance metrics beyond raw throughput.

The significance of this research lies in its practical implications for deploying LLMs at scale. By highlighting the pitfalls of current benchmarking practices and unveiling the often-hidden costs of compression—such as increased end-to-end latency due to varied response lengths and the generation of "negative samples" (accuracy degradation)—the authors provide crucial insights for practitioners. Furthermore, they propose a set of simple yet effective tools, including throughput predictors, length predictors, and negative sample evaluators, designed to facilitate more informed decision-making and robust deployment of KV cache compression strategies in production settings. This work serves as a vital guide for optimizing LLM serving efficiency without compromising reliability or user experience.

Background

▶ Watch: Introduction to KV cache compression for LLM serving (0:00)

The rapid advancement and widespread adoption of Large Language Models have introduced significant computational and memory challenges, particularly during the inference phase. One of the most prominent bottlenecks is the Key-Value (KV) cache. In the transformer architecture, during token generation (decoding), the key and value states from previous tokens are stored in memory to avoid recomputing them for each new token. This cache, often referred to as the KV cache, grows linearly with the sequence length and batch size. For very long contexts and large batch sizes, the KV cache can consume several hundreds of gigabytes of GPU memory, making it the dominant memory consumer and a major factor limiting the throughput and maximum sequence length an LLM server can handle.

To mitigate this memory pressure, researchers have explored two primary categories of KV cache compression techniques:

1. Quantization-based methods: These techniques reduce the precision of the KV cache tensors, typically converting them from FP16 or BF16 to lower-bit integer formats (e.g., INT8, INT4, or even INT2). The goal is to save memory while minimizing accuracy loss. Examples include dynamic quantization schemes or techniques that introduce an extra step to handle quantization outliers.
2. Sparsity-based methods: These approaches aim to identify and remove or offload less important parts of the KV cache. This can be done at various granularities, such as token level, layer level, head level, or channel level. The underlying assumption is that not all KV cache entries are equally critical for maintaining model accuracy, and significant memory savings can be achieved by pruning or moving redundant information to slower memory tiers.

Despite substantial research efforts in both directions, the practical applicability of these compression algorithms in real-world production environments remains a significant concern. Many existing studies often evaluate performance in idealized settings, overlooking crucial aspects of modern LLM serving systems. For instance, common benchmarks frequently neglect the integration with high-performance acceleration techniques like FlashAttention and PageAttention, which are standard in production frameworks. This disconnect leads to a gap between reported research efficacy and actual deployment challenges, where factors such as computational overhead from irregular memory access patterns, compatibility issues with system-level optimizations, and the impact on end-to-end latency and response quality are often not adequately addressed. This paper systematically investigates these overlooked aspects, providing a much-needed empirical and analytical perspective on the true costs and benefits of KV cache compression.

Key Findings

▶ Watch: Overview of two main KV cache compression approaches (2:48)

The comprehensive literature review and quantitative analysis conducted in this study reveal several critical findings, challenging common assumptions about KV cache compression efficacy.

Qualitative Observations from Literature Review (40+ papers surveyed):

1. Quantization-based Techniques:

• Irregular Computational Patterns: Finer-grained quantization, while potentially preserving accuracy better, often introduces irregular computational patterns. This can severely limit the effective utilization of GPU resources, negating the memory savings with reduced computational efficiency.
• Outlier Mitigation Overhead: Some methods attempt to mitigate quantization outliers, but this typically incurs additional GPU memory consumption and computational overhead, potentially offsetting any efficiency gains.
• Compatibility with PageAttention: Many quantization algorithms employ a window-based approach, keeping recent KV cache entries at full precision for accuracy. This design complicates compatibility with PageAttention, which requires uniform tensor management. Managing two types of page tensors (full precision and quantized) introduces unstructured computation patterns, increasing deployment complexity and degrading computational efficiency.

1. Sparsity-based Techniques:

• Granularity vs. Overhead: Similar to quantization, pursuing finer granularity (e.g., token, layer, head, channel level) in sparsity methods can yield improved accuracy but at the cost of high computational overhead due to irregular memory access and computation patterns.
• Neglect of Modern Serving Techniques: Many sparsity studies fail to consider important serving techniques like FlashAttention and PageAttention when measuring throughput. Integrating sparsity methods with these highly optimized techniques can significantly increase implementation complexity and undermine potential computational efficiency advantages, highlighting the need for dedicated system-level optimizations.

1. Evaluation Settings in Prior Work:

• Unreliable Throughput Benchmarking: Many compression studies use the Transformers library (TRL) for throughput measurement, often omitting crucial LLM acceleration techniques like FlashAttention or PageAttention. This leads to unreliable results that do not reflect production environments.
• Overlooked Response Length Impact: Prior work often neglects how compression affects response lengths, a critical factor for end-to-end latency performance.
• Long Context and Negative Samples: Long context tasks pose significant challenges for KV cache compression algorithms to maintain accuracy. Many studies report overall performance but overlook the analysis of response quality for individual samples, especially the occurrence of "negative samples" (samples where accuracy significantly degrades).

Quantitative Analysis and Empirical Results:

1. Throughput Measurement Reliability: The study empirically demonstrates that throughput results obtained using the Transformers library (TRL) are not reliable for production environments. PageAttention and FlashAttention significantly improve decoding throughput, and relative speedups measured on TRL do not translate effectively when prominent techniques are used. It is crucial to measure throughput using established LLM serving frameworks that integrate these modern acceleration techniques.
2. Conditional Efficiency of Compression: KV cache compression methods exhibit negative computational efficiency in certain scenarios, specifically with particular batch sizes, sequence lengths, and during either the prefill or decoding stage. The authors recommend applying compression algorithms primarily for serving requests with heavy KV caches, where memory savings are substantial enough to outweigh potential computational overheads.
3. Impact on Response Lengths and End-to-End Latency: Lossy KV cache compression tends to significantly increase the variation in response lengths, a phenomenon that worsens with higher compression ratios. This elongation of responses can lead to longer end-to-end latencies, potentially offsetting any throughput benefits gained from memory compression. The study found that performance gains of compression methods on end-to-end latency are often not significant, and can even be negative.
4. Prevalence of Negative Samples: KV cache compression algorithms naturally produce negative samples, defined as benign samples where compression causes a relative accuracy loss beyond a set threshold (e.g., 10%). While algorithms with higher intrinsic accuracy can reduce their occurrence, completely eliminating them remains challenging.
5. Task-Specific Vulnerability: The analysis revealed that KV cache compression algorithms struggle particularly with specific tasks like summarization and question answering. These tasks heavily rely on accurate context information, making them highly susceptible to accuracy degradation from compression.

In summary, the key findings underscore that while KV cache compression offers memory benefits, its application requires careful consideration of its complex interactions with modern LLM serving frameworks, its impact on overall latency, and its potential to degrade accuracy for critical tasks.

Technical Deep Dive

▶ Watch: Qualitative challenges of sparsity-based compression methods (6:10)

This work provides a critical technical assessment rather than introducing a new compression algorithm. The deep dive focuses on the methodology for evaluating existing techniques and the proposal of practical tools to address their limitations in production.

Evaluation Setup and Frameworks:

The study employed a robust evaluation setup designed to reflect real-world LLM serving conditions.

• Frameworks: Two primary frameworks were used:
• The Transformers library (TRL): Used for comparative analysis, particularly to highlight its limitations in representing production performance due to its common omission of advanced acceleration techniques.
• LMDeploy: Chosen for its ease of modification and its native support for crucial LLM acceleration techniques, specifically FlashAttention and PageAttention. This allowed for a more realistic assessment of compression techniques in a production-like environment.
• Large Language Models (LLMs): The evaluation primarily focused on:
• Llama: Specifically, the Llama 7B model was used for detailed analysis of prefill and decoding throughput.
• Mixtral: Also included in the evaluation, indicating a broader applicability of the findings.
• Datasets:
• SharedGPT: Utilized for analyzing response length distributions and end-to-end latencies.
• LongBench: Employed for conducting the negative samples analysis, particularly for long-context scenarios.
• Compression Algorithms Evaluated: The study selected a representative set of state-of-the-art KV cache compression techniques:
• Quantization-based techniques:
• KVQuant: A prominent quantization method for KV caches.
• Gear: Another representative quantization approach.
• Sparsity-based techniques:
• H2O: A sparsity method that identifies and prunes less important KV cache entries.
• StreamingLLM: A method designed for efficient long-context inference by maintaining a small, fixed-size attention window.

Challenges in Integrating Compression with Modern Techniques:

A significant technical insight is the inherent difficulty in integrating many existing compression methods with highly optimized LLM serving techniques.

• FlashAttention: This technique reorders attention computation to reduce memory I/O, but it often requires contiguous memory layouts or specific data structures. Irregular access patterns introduced by sparsity or certain quantization schemes can hinder its benefits or require complex re-engineering.
• PageAttention: This technique manages KV cache memory in pages, similar to virtual memory, enabling efficient memory sharing across requests and reducing fragmentation. Window-based quantization, which keeps a portion of the KV cache at full precision and the rest quantized, demands two distinct types of page tensors. This creates an unstructured computation pattern when calculating attention outputs, degrading computational efficiency and increasing deployment complexity.

Proposed Tools for Robust Deployment:

Recognizing the limitations of existing methods and the complexities of their deployment, the authors propose three practical tools designed to enable more effective and informed use of KV cache compression in production:

1. Throughput Predictor:

• Mechanism: The observation is that KV cache compression primarily impacts attention throughput. This tool works by profiling the attention throughput across various batch sizes and KV cache lengths.
• Functionality: It uses these profiles to predict the overall prefill and decoding throughput for different serving scenarios, allowing operators to understand the real-time performance implications of applying compression. This helps in dynamically deciding when and where to apply compression.

1. Length Predictor:

• Mechanism: This tool addresses the issue of increased response length variability. It collects response lengths generated by different KV cache compression algorithms.
• Functionality: A BERT-based classifier is trained on this data to predict output length ranges. By anticipating how compression might alter response lengths, serving systems can better manage resource allocation and estimate end-to-end latencies, potentially mitigating the negative impact of elongated responses.

1. Negative Sample Evaluator:

• Mechanism: This tool directly tackles the problem of accuracy degradation. It establishes a specific threshold (e.g., 10% relative accuracy loss) to detect "negative samples."
• Functionality: It incorporates this detection into a benchmark to quantify the proportion of negative samples across different task types. The empirical results demonstrated significant drops in accuracy for tasks like summarization and question answering. This evaluator stresses the importance of developing more robust solutions and provides a metric to monitor the reliability of compressed models, especially for critical applications.

Request Routing Showcase (Conceptual):

While not detailed in the talk, the authors briefly mention a "showcase of using the tools in request routing in serving." This implies an intelligent system where the predictors and evaluator can dynamically inform a router to decide whether to serve a specific request using a compressed KV cache or a full-precision one, based on the predicted throughput, latency, and potential for accuracy loss for that request's characteristics. This intelligent routing mechanism would allow for optimized resource utilization while maintaining performance and quality of service.

These tools collectively offer a framework for practical, data-driven decision-making when deploying KV cache compression, moving beyond simplistic memory-saving metrics to a holistic view of end-to-end performance and reliability.

Experimental Setup & Results

▶ Watch: Analysis of pitfalls in existing KV cache compression evaluations (8:10)

The experimental setup was designed to provide a realistic assessment of KV cache compression techniques, moving beyond idealized benchmarks often found in academic literature. The key components and findings are detailed below:

Experimental Setup:

• Frameworks: The evaluation primarily leveraged LMDeploy for its robust support of modern LLM acceleration techniques, including FlashAttention and PageAttention, which are crucial for production environments. The Transformers library (TRL) was also used as a baseline to demonstrate the disparity between its benchmarks and production-realistic measurements.
• Models: The primary LLMs studied were Llama 7B and Mixtral, representing widely used and structurally diverse models.
• Datasets:
• SharedGPT: Employed to analyze the distribution of response lengths and end-to-end latencies across various compression methods.
• LongBench: Used for a thorough investigation into the occurrence and impact of "negative samples," particularly in long-context scenarios.
• Compression Algorithms: A selection of state-of-the-art algorithms was chosen for evaluation:
• Quantization-based: KVQuant and Gear.
• Sparsity-based: H2O and StreamingLLM.
• Metrics: The evaluation focused on a comprehensive set of metrics:
• Throughput: Measured during both prefill and decoding stages, considering different batch sizes and KV cache lengths.
• Response Length Distribution: Analyzing how compression affects the variability and average length of generated outputs.
• End-to-End Latency: Measuring the total time from request initiation to final token generation for individual samples.
• Negative Sample Proportion: Quantifying the percentage of samples where compression leads to a significant degradation in accuracy (defined as a relative accuracy loss beyond a 10% threshold).

Headline Numbers & Key Results:

1. Impact of Modern Acceleration Techniques on Throughput:

• Figures A and B (not explicitly shown in the transcript but referred to) demonstrated that PageAttention and FlashAttention significantly improve decoding throughput, confirming their essential role in efficient LLM serving.
• Crucially, Figures C and D highlighted that the relative speedup observed when using the Transformers library (TRL) was not significant when measured against setups incorporating PageAttention and FlashAttention. This strongly suggests that throughput results obtained solely on TRL are unreliable for production deployment, underscoring the need for realistic benchmarking.

1. Conditional Efficiency of KV Cache Compression:

• The experiments revealed that KV cache compression methods often show negative computational efficiency in certain scenarios. This means that for specific combinations of batch sizes, sequence lengths, and at different stages (prefill vs. decoding), the overhead introduced by compression (e.g., irregular computation, dequantization) can outweigh the benefits of reduced memory footprint, leading to lower effective throughput.
• Recommendation: Based on these findings, the authors recommend applying compression algorithms primarily for serving requests with heavy KV caches, where the memory savings are most critical and substantial enough to potentially justify or overcome the associated computational overhead.

1. Response Length Variation and End-to-End Latency:

• A significant finding was that lossy compression techniques tend to make response length variations much larger. This effect becomes more pronounced with higher compression ratios.
• While compression might reduce memory, leading to theoretical throughput increases, the increased response lengths can lead to longer end-to-end latencies. This extended latency can effectively outweigh the benefits of raw throughput gains, making the overall user experience worse.
• Measurements of end-to-end latencies for individual samples in the SharedGPT dataset (with batch size 1) showed that the performance gains from compression methods were often not significant and could even be negative due to reduced throughput and increased response lengths.

1. Occurrence of Negative Samples:

• The study confirmed that KV cache compression algorithms naturally produce negative samples. These are instances where the model's output quality degrades beyond an acceptable threshold due to the compression.
• While algorithms designed for higher accuracy can reduce the frequency of negative samples, it remains challenging to eliminate them entirely.
• Task-Specific Vulnerability: Setting a 10% accuracy loss threshold, the analysis (left bar chart mentioned in transcript) investigated the proportion of negative samples across different task types. It was observed that these compression algorithms particularly struggle with tasks like summarization and question answering. This is attributed to the high reliance of these tasks on accurate and complete contextual information, which is more susceptible to degradation from lossy compression.

These results collectively highlight that KV cache compression is a nuanced optimization. Its benefits are not universal, and its application requires a thorough understanding of its impact on computational efficiency, end-to-end latency, and, critically, the quality of generated outputs for specific tasks.

Practical Implications

▶ Watch: Lossy compression impacts response length and end-to-end latency (14:00)

The findings of this comprehensive study carry significant practical implications for anyone involved in designing, deploying, or managing large language model serving infrastructure.

1. Rethink Benchmarking Practices:

• For Practitioners & Infra Teams: Do not rely solely on throughput numbers reported using the Transformers library (TRL) for production deployment decisions. These benchmarks often omit critical acceleration techniques like FlashAttention and PageAttention, leading to inflated or misleading performance expectations. Always evaluate compression algorithms within a production-realistic framework (e.g., LMDeploy) that incorporates these advanced optimizations.
• For Model Builders: When proposing new compression algorithms, ensure your evaluation methodology is robust, including integration with state-of-the-art serving techniques to demonstrate real-world applicability.

1. Strategic Application of Compression:

• For Infra Teams & Deployers: KV cache compression is not a universal solution. The study clearly shows that it can lead to negative computational efficiency in certain scenarios. It should be applied judiciously, primarily for requests that generate heavy KV caches (e.g., very long prompts, large batch sizes), where memory savings are the most critical bottleneck. For lighter loads, the overhead might outweigh the benefits.
• Tradeoffs: Understand that compression introduces a complex set of tradeoffs: memory savings vs. computational overhead, raw throughput vs. end-to-end latency, and efficiency vs. accuracy. There is no one-size-fits-all answer.

1. Holistic Performance Evaluation:

• For Deployers: Beyond raw throughput, meticulously evaluate the impact of compression on end-to-end latency. Throughput gains can be entirely negated by increased response lengths, leading to a poorer user experience. Monitor both metrics.
• For Model Builders: Consider the entire inference pipeline, from prompt reception to full response generation, rather than just isolated components.

1. Accuracy and Reliability for Critical Tasks:

• For Practitioners & Model Builders: Be highly cautious when applying KV cache compression to tasks sensitive to contextual accuracy, such as summarization and question answering. These tasks are particularly vulnerable to negative samples (accuracy degradation).
• Mitigation: Implement safeguards, such as routing critical requests to uncompressed paths or using the proposed Negative Sample Evaluator to monitor output quality. Developing more robust compression solutions that preserve accuracy for these tasks is a key area for future research.

1. Leveraging Analytical Tools for Intelligent Deployment:

• For Infra Teams: The proposed Throughput Predictor, Length Predictor, and Negative Sample Evaluator are invaluable tools. They enable data-driven decision-making for dynamic request routing, allowing systems to intelligently decide whether to apply compression based on predicted performance and accuracy for specific incoming requests. This shifts from a static "on/off" compression approach to a more adaptive, optimized strategy.
• For Developers: Incorporating these types of analytical components into LLM serving frameworks can significantly enhance their robustness and efficiency.

In essence, this work provides a much-needed reality check for KV cache compression. It moves the discussion from theoretical memory savings to practical deployment challenges, emphasizing the need for comprehensive evaluation, strategic application, and intelligent management to truly unlock the efficiency benefits of compressed LLM serving.

Key Takeaways

• Existing KV cache compression benchmarks often provide misleading performance figures for production environments by neglecting modern LLM acceleration techniques like FlashAttention and PageAttention.
• KV cache compression is not universally beneficial; its efficacy depends heavily on specific serving scenarios (e.g., batch size, sequence length), and it can even lead to negative computational efficiency in certain cases.
• Lossy compression techniques significantly increase the variability of response lengths, which can lead to longer end-to-end latencies that effectively negate any throughput gains from memory reduction.
• Compression algorithms inherently introduce "negative samples" (instances of significant accuracy loss), particularly impacting context-sensitive tasks such as summarization and question answering.
• A comprehensive evaluation framework for KV cache compression must consider not only throughput and memory but also response length distribution, end-to-end latency, and the prevalence of negative samples.
• The proposed analytical tools—Throughput Predictor, Length Predictor, and Negative Sample Evaluator—are crucial for enabling intelligent, robust, and adaptive deployment of KV cache compression in real-world LLM serving systems.

About the Speaker(s)

This work, "Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving," is a collaborative effort by Gao Wei, Xinyu Zhou, Peng Sun, Tianwei Zhang, and Yonggang Wen. The research was conducted jointly by Nanyang Technological University, S Lab, and Shanghai AI Lab. The presentation at MLSys 2025 was delivered by Hanyu from Alibaba, who stepped in on behalf of the first author, Gao Wei, due to unforeseen visa issues. While Hanyu presented the work, the core intellectual contributions and research were led by the named authors.