HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression

HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression

Speakers: Yujin Wang, Shunan Dong, Yichen You, Huazhong Yang, Yongpan Liu, Hongyang Jia

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

The talk "HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression" by Yujin Wang and colleagues from Tsinghua University addresses a critical bottleneck in the on-device fine-tuning of large language models (LLMs): the prohibitive memory consumption of buffer activations. While parameter-efficient fine-tuning (PEFT) methods like LoRA (Low-Rank Adaptation) and QLoRA have significantly reduced memory overhead associated with model weights and optimizer states, the memory footprint of activations required for backward propagation has emerged as the new dominant burden. HyC-LoRA proposes a novel, systematic approach to compress these buffer activations through a hybrid compression mechanism tailored to different activation types and distributions.

This work is particularly significant in the context of expanding LLM capabilities to edge devices and personal computing environments. Enabling efficient on-device fine-tuning is crucial for privacy, flexibility, and real-time adaptation of models for specific tasks such as personalized chatbots or robotic environment interaction. By achieving substantial memory reductions without compromising accuracy or throughput, HyC-LoRA paves the way for broader deployment and customization of powerful LLMs on resource-constrained hardware, democratizing access to advanced AI capabilities.

Background

▶ Watch: Introduction to on-device fine-tuning challenges and memory burden (0:00)

The rapid advancements in large language models (LLMs) have demonstrated their immense potential across a myriad of applications. However, to harness their full power for specialized tasks and domains, these foundation models often require fine-tuning on custom datasets. This process allows models to adapt to specific user preferences or environmental nuances, moving beyond their generalized pre-training. The demand for on-device fine-tuning is growing due to benefits like enhanced data privacy, reduced latency, and increased operational flexibility, eliminating the need to transfer sensitive data to cloud servers.

Despite the advantages, fine-tuning LLMs, especially on resource-constrained devices, presents significant memory challenges. A direct comparison between inference and fine-tuning data flows highlights this issue. While inference primarily involves forward propagation, fine-tuning additionally necessitates storing buffer activations for backward computation and optimizer states for weight updates. These extra memory requirements often make on-device deployment infeasible for current LLM architectures.

Previous research has largely focused on alleviating memory pressure by compressing model weights and optimizer states. Methods like LoRA (Low-Rank Adaptation) reduce the number of tunable parameters by injecting small, low-rank matrices into existing model layers, thereby significantly decreasing the memory footprint of the weights themselves and their corresponding optimizer states. Further advancements, such as QLoRA, introduced 4-bit quantization for the base model weights, pushing these memory savings even further. However, despite these innovations, the memory occupied by buffer activations—intermediate outputs of neural network layers that must be retained for gradient calculation during backpropagation—remained largely unaddressed. The speaker emphasizes that "parameter efficient is not exactly equivalent to memory efficient" because even with reduced tunable parameters, the full set of buffer activations for the entire computation graph still needs to be stored. This oversight has made uncompressed buffer activations the "new burden of the system," often consuming the major share of system memory during fine-tuning. Experiments have also revealed that linear buffer activations are particularly critical for compression, as errors introduced during their compression can propagate significantly during backward passes, impacting model convergence and accuracy. This context sets the stage for HyC-LoRA, which specifically targets this overlooked memory bottleneck.

Key Findings

▶ Watch: HyC-LoRA: hybrid activation compression for all fine-tuning components (4:00)

HyC-LoRA introduces a comprehensive solution for memory-efficient LoRA fine-tuning by systematically addressing the buffer activation bottleneck. The core findings and contributions are:

• Hybrid Activation Compression: HyC-LoRA develops a novel hybrid compression mechanism tailored for different types and distributions of buffer activations (specifically focusing on linear activations). This approach strategically applies quantization techniques, balancing compression ratio with accuracy requirements.
• Intra-Operator Optimization for Outliers: The method identifies and handles structured outliers within linear buffer activations (e.g., from RMSNorm and LayerNorm layers). By detecting these outliers during a calibration phase and storing them in full precision while quantizing the rest, HyC-LoRA significantly reduces compression error propagation with negligible overhead.
• Inter-Operator Optimization via LoRA Reorder Computing: HyC-LoRA proposes a LoRA reorder computing method that leverages the structure of LoRA adapters to optimize memory and computation. It strategically recomputes parts of the LoRA representation during backward propagation, keeping critical information in full precision without additional memory cost. This is further enhanced by kernel fusion, which integrates compression/decompression with other element-wise operations.
• Significant Memory Reduction: At the system level, HyC-LoRA achieves an impressive up to 8.2 times buffer memory reduction ratio. When considering all components (weights, optimizer, and activations), the end-to-end memory reduction ranges from 1.57 times to 4.1 times, depending on experimental settings.
• Comparable Accuracy and Throughput: Despite substantial memory savings, HyC-LoRA maintains fine-tuning accuracy comparable to uncompressed baselines. Furthermore, due to optimized operator implementation and reduced memory access, it demonstrates throughput comparable to uncompressed baselines and is 1.1 times to 1.54 times faster than traditional gradient checkpointing methods, and also outperforms some previous compression-based methods that suffer from irregular memory access.
• Compatibility and Flexibility: HyC-LoRA is logically orthogonal to gradient checkpointing, allowing for their combined use to achieve even higher memory efficiency, particularly beneficial for long sequence fine-tuning scenarios. It can also be flexibly integrated with other selective recomputation methods.

In essence, HyC-LoRA advances the state-of-the-art in memory-efficient fine-tuning by systematically tackling the previously unaddressed challenge of buffer activation memory, enabling the deployment of LLMs on a wider range of hardware platforms.

Technical Deep Dive

▶ Watch: Intra-operator compression: tackling structured outliers in activations (7:00)

HyC-LoRA's technical prowess lies in its multi-faceted approach to compressing buffer activations, integrating hybrid precision, outlier handling, and optimized computational reordering. The primary focus is on linear buffer activations, which are identified as particularly critical due to their role in error propagation during backpropagation.

Hybrid Compression Mechanism for Linear Buffer Activations

The core idea for linear buffer activations is a hybrid precision setting. During the forward pass, large chunks of buffer activations are quantized, while smaller, more critical chunks are kept in full precision. For the backward pass, the quantized activations are dequantized for subsequent calculations.

• Quantization Granularity: The quantization is performed per-channel, which allows for a finer-grained adaptation to the varying distributions across different channels of the activation tensors.
• Calibration Stage: To determine the quantization parameters (specifically, the scaling factor), HyC-LoRA introduces a brief calibration stage. This stage, lasting only a few steps, statically analyzes the data distribution of buffer activations. The key insight here is that for fine-tuning scenarios (as opposed to training from scratch), the distribution of buffer activations is relatively stable. Therefore, calculating and fixing scaling factors during calibration avoids the overhead of dynamic statistics computation during the main training phase.
• Training Stage: Once calibrated, the scaling factors are frozen and directly applied to compute compressed tensors throughout the training process.

Intra-Operator Optimization: Structured Outlier Detection

Further enhancing the compression of linear buffer activations, HyC-LoRA employs an intra-operator compression strategy, specifically targeting structured outliers within certain layers.

• Observation: The speakers observed that buffer activations from RMSNorm and LayerNorm layers exhibit "structured outliers" in specific channels. These channels, while having small overall norms, significantly increase compression errors if treated uniformly.
• Outlier Detection: During the calibration stage, HyC-LoRA calculates the channel-wise norm for these activations. It then identifies and marks the top 10% of channels with the highest norms as outliers.
• Hybrid Storage: During the training stage, these marked "outlier channels" are stored in full precision with regular memory access. The remaining 90% of channels are then quantized. In the backward pass, these two parts are merged for computation.
• Negligible Overhead: The memory overhead for storing the indexes of these outlier channels is negligible (typically less than 1% of the total memory). This targeted approach prevents significant error propagation while maintaining high compression ratios. For other linear buffer activations that do not exhibit such clear outlier features, the general heterogeneous precision settings (quantized big chunk, full precision small chunk) are applied.

Inter-Operator Optimization: LoRA Reorder Computing and Kernel Fusion

To minimize the overhead associated with compression/decompression and to improve overall throughput, HyC-LoRA introduces inter-operator optimizations.

• LoRA Reorder Computing: This method logically rearranges the order of operations during forward and backward propagation, specifically leveraging the structure of LoRA adapters.
• Forward Propagation: Only the "mean part" of the buffer activations (i.e., the activations from the base model without LoRA adjustments) is dequantized. The information related to the LoRA adapter (denoted as YAB in the talk, representing the output of X A B where X is input, A and B are LoRA matrices) is retained in full precision.
• Backward Propagation: Instead of storing YAB in full precision, its value is recomputed by multiplying the dequantized XA (from the mean part buffer activation) with the LoRA adapter B, and then adding it to the mean part. This reconstructed, full-precision YAB is then used for the backpropagation of linear operators. This ingenious reordering ensures that the critical LoRA information is always available in high precision without incurring additional memory overhead for storing it.
• Kernel Fusion: To further optimize performance and reduce memory access overhead, HyC-LoRA implements custom CUDA kernels. These kernels fuse the LoRA reorder computing logic with the compress and decompress processes, as well as other element-wise operations like activation functions.
• Advantages: This fusion brings two key benefits:

1. It reduces the number of intermediate tensors that need to be explicitly compressed and de-compressed, thereby lowering the computational overhead of these operations.
2. It minimizes external memory access, as multiple operations can be performed directly within GPU registers or shared memory, leading to higher fine-tuning speeds.

In summary, HyC-LoRA combines a sophisticated hybrid quantization scheme with intelligent outlier handling and a novel reordering of LoRA computations, all orchestrated through optimized kernel implementations. This holistic approach ensures significant memory savings while preserving model accuracy and maintaining high training throughput.

Experimental Setup & Results

▶ Watch: LoRA reorder computing: hybrid precision for adapter activations (16:00)

The HyC-LoRA team conducted extensive experiments at both the algorithmic and system levels to validate their proposed method. All system-level experiments were performed on single devices, covering a range of hardware platforms from edge devices to cloud-grade GPUs, and did not consider distributed frameworks.

Algorithmic Level Experiments

At the algorithmic level, HyC-LoRA's fine-tuning performance was evaluated across:

• Different model sizes: Ensuring scalability and effectiveness for various LLM scales.
• Different model architectures: Testing generalization beyond a single model family.
• Various tasks: Demonstrating applicability across diverse downstream applications.

The results consistently showed that HyC-LoRA's fine-tuning accuracy is compatible with the baseline (uncompressed) methods. This indicates that the aggressive memory compression strategies do not lead to significant performance degradation, a crucial aspect for practical deployment. Comparisons with "other lightweight methods" revealed that HyC-LoRA "advances the operator frontier of the memory efficient fine-tuning between the compression ratio and accuracy," suggesting it achieves a superior balance compared to existing techniques.

System Level Experiments

System-level evaluations focused on quantitative metrics such as memory consumption and training throughput, critical for practical deployment.

• Memory Consumption:
• Buffer Memory Reduction: HyC-LoRA achieved an impressive up to 8.2 times reduction in buffer memory consumption. This directly addresses the main bottleneck identified by the authors.
• End-to-End Memory Reduction: When considering all components involved in fine-tuning—including weights, optimizer states, and buffer activations—HyC-LoRA delivered an end-to-end memory reduction ratio ranging from 1.57 times to 4.1 times. This variability depends on the specific experimental settings, such as model size, batch size, and the proportion of memory consumed by different components. These numbers are significant for enabling fine-tuning on devices with limited total memory.

• Training Throughput:
• Comparable to Baseline: Despite the additional operations for compression and decompression, HyC-LoRA's throughput was found to be comparable to uncompressed baseline methods. This is attributed to its small computation overhead and the extensive operator optimizations, including kernel fusion, which minimize the performance impact.
• Superior to Gradient Checkpointing: HyC-LoRA demonstrated a 1.1 times to 1.54 times throughput boost compared to traditional gradient checkpointing methods. Gradient checkpointing reduces memory by recomputing activations during the backward pass, but this comes at the cost of increased computational time. HyC-LoRA's direct compression, coupled with efficient implementation, proves more performant in terms of speed while still offering substantial memory benefits.
• Outperforming Previous Compression Methods: The method was also found to be faster than some previous compression-based techniques that suffer from irregular memory access patterns, highlighting the benefit of HyC-LoRA's structured outlier handling and fused kernels.

These experimental results unequivocally demonstrate HyC-LoRA's ability to provide substantial memory savings for LoRA fine-tuning without compromising accuracy or significantly impacting training speed, making it a highly practical solution for on-device LLM adaptation.

Practical Implications

▶ Watch: Experimental results demonstrate HyC-LoRA's accuracy and memory efficiency (19:00)

HyC-LoRA's advancements have profound practical implications for practitioners, infrastructure teams, model builders, and deployers working with large language models, especially in resource-constrained environments.

First and foremost, HyC-LoRA significantly expands the feasibility of on-device and edge fine-tuning of LLMs. By drastically reducing the memory footprint of buffer activations – the previously unaddressed bottleneck – it makes it possible to adapt powerful models on hardware that was previously insufficient. This is critical for applications demanding high privacy (e.g., personal data processing), low latency (e.g., real-time robotic control), or operation in disconnected environments. Infra teams can now consider deploying fine-tuning capabilities on a wider range of devices, from embedded systems to smartphones and local workstations, reducing reliance on expensive cloud GPUs.

For model builders, HyC-LoRA offers a new tool to optimize their fine-tuning workflows. The ability to achieve substantial memory savings (up to 8.2x for buffer memory and 1.57x to 4.1x end-to-end) without sacrificing accuracy means that larger models or larger batch sizes can be fine-tuned on existing hardware, or the same fine-tuning tasks can be accomplished with less powerful (and thus cheaper or more energy-efficient) hardware. This can lead to reduced operational costs and a lower carbon footprint for AI development.

The work also clarifies important tradeoffs and limitations. Unlike some lightweight methods that simplify or truncate the backpropagation compute graph (thereby reducing both buffer activations and backward calculations), HyC-LoRA does not inherently improve the speed of a single iteration by reducing backward computation flow. However, by maintaining the full backward computation graph, it ensures that the convergence rate of training is preserved, which can be a significant advantage over methods that might accelerate individual iterations but slow down overall training convergence due to approximation errors. This distinction is crucial for deployers who prioritize robust training outcomes.

Furthermore, HyC-LoRA exhibits excellent compatibility with existing optimization techniques. It is highlighted as logically orthogonal to gradient checkpointing, meaning these two methods can be stacked to achieve even higher memory efficiency. This is particularly valuable in scenarios involving long sequence fine-tuning, where both buffer activations and the overall computation graph can become excessively large. The flexibility to integrate with selective recomputation methods further enhances its utility, allowing practitioners to combine HyC-LoRA with other techniques as needed for specific use cases.

Finally, the talk emphasizes that HyC-LoRA specifically focuses on the unique memory consumption aspects of the training process, particularly buffer activations that must be kept for backward propagation. This differentiates it from inference engines, where activations can often be dropped after a layer's computation (except for KV cache in transformers). This clear focus helps practitioners understand where HyC-LoRA offers the most benefit and how it complements existing inference optimization strategies.

Key Takeaways

• Buffer Activations are the New Bottleneck: While LoRA and QLoRA optimize weights and optimizers, uncompressed buffer activations became the dominant memory burden for LLM fine-tuning, especially on-device.
• Hybrid Compression for Activations: HyC-LoRA introduces a novel hybrid compression mechanism for linear buffer activations, combining quantization with full-precision for critical parts, tailored by per-channel granularity and a calibration stage.
• Outlier Handling and LoRA Reorder Computing: Specific intra-operator optimizations (structured outlier detection in RMSNorm/LayerNorm) and inter-operator optimizations (LoRA reorder computing and kernel fusion) significantly enhance accuracy and throughput.
• Dramatic Memory Savings: The method achieves up to 8.2x buffer memory reduction and 1.57x to 4.1x end-to-end memory reduction, enabling fine-tuning on resource-constrained devices.
• No Performance Compromise: HyC-LoRA maintains accuracy comparable to baselines and delivers throughput comparable to uncompressed methods, even outperforming gradient checkpointing (1.1x to 1.54x faster).
• Stackable and Flexible: It is logically orthogonal to gradient checkpointing, allowing for combined use, and can integrate with other selective recomputation methods for maximum efficiency.

About the Speaker(s)

The primary speaker for "HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression" was Yujin Wang. He is part of a research team from Tsinghua University, which is credited as the institution behind this work. The co-authors listed are Shunan Dong, Yichen You, Huazhong Yang, Yongpan Liu, and Hongyang Jia. The talk highlights their collective expertise in addressing complex challenges in AI/ML systems, particularly in optimizing large models for practical deployment on diverse hardware platforms.