From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control

From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control

Speakers: Chi Zhang, REN Lianhai, Jingpu Cheng, Qianxiao Li

Conference: ICML 2025

YouTube: https://slideslive.com/39044068

Overview

This talk, presented by Chi Zhang and co-authors REN Lianhai, Jingpu Cheng, and Qianxiao Li from the National University of Singapore, introduces a novel perspective on Parameter-Efficient Fine-Tuning (PEFT) algorithms, particularly Low-Rank Adaptation (LoRA). Moving beyond the conventional understanding of LoRA as a weight-tuning technique, the authors propose a reinterpretation through the lens of classical control theory. This shift in perspective leads to the development of a state-based fine-tuning approach utilizing parallel control, which significantly reduces memory consumption and training time for large language models.

The motivation for this work stems from the increasing memory demands of fine-tuning large pre-trained models, which often limits the accessibility of such tasks to specialized hardware. By reframing LoRA's mechanism as a control system perturbing intermediate states rather than modifying weights, the proposed method unlocks substantial memory savings. This innovation is particularly impactful as it enables the fine-tuning of models like Llama-2 7B and Llama-3 8B on consumer-grade NVIDIA 3090 GPUs, democratizing access to powerful AI models for a broader range of researchers and practitioners. The talk highlights a unique theoretical understanding that not only simplifies analysis but also delivers tangible practical benefits in terms of resource efficiency and computational speed.

Background

▶ Watch: Introduction: LoRA and its weight-tuning perspective (0:00)

The landscape of modern machine learning, especially in large language models (LLMs), is dominated by the need for efficient adaptation of massive pre-trained models to downstream tasks. Parameter-Efficient Fine-Tuning (PEFT) methods have emerged as a critical solution to this challenge, addressing the prohibitive computational and memory costs associated with full fine-tuning. Among these, Low-Rank Adaptation (LoRA) stands out as one of the most widely adopted and effective techniques.

Conventionally, LoRA is understood as a weight-tuning approach. The prevailing narrative suggests that LoRA works by inserting small, low-rank matrices—typically denoted as A and B—in parallel to the existing weight matrices of a pre-trained model. These low-rank matrices are then fine-tuned, capturing "useful adaptation signals" while keeping the original, large pre-trained weights frozen. This design makes fine-tuning parameter-efficient and helps preserve the original knowledge embedded in the base model. This core idea has inspired a family of LoRA variants, including DoRA, PiSSA, RS-LoRA, VeRA, and HyLoRA, all largely adhering to this weight-tuning paradigm.

However, the talk challenges this conventional wisdom by presenting counter-examples and alternative interpretations that suggest the weight-tuning view might not be the complete or even the most accurate explanation for PEFT's efficacy. One prominent counter-example is the Adapter method, proposed as early as 2019. Adapters also introduce low-rank matrices, but critically, they insert them sequentially between transformer layers (e.g., before and after MLP layers). In this configuration, these low-rank matrices do not directly modify or update any existing weight matrices; instead, they are "plugged into the forward pass" to process intermediate states. This sequential insertion fundamentally differs from LoRA's parallel weight modification, yet it achieves strong performance, calling into question the exclusivity of the weight-tuning hypothesis.

Further evidence against a solely weight-based understanding comes from the speakers' own prior work, a 2022 paper on "Parameter-Efficient Fine-Tuning with Control." In that research, they moved beyond low-rank modules entirely, employing more flexible blocks like attention or residual blocks. Despite the absence of explicit low-rank structures, these methods still exhibited very strong performance. These examples collectively lead to the fundamental question posed by the talk: are these weight fine-tuning views the only or correct explanation for the success of PEFT algorithms? This critical inquiry sets the stage for a paradigm shift, proposing a new theoretical framework rooted in control systems.

Key Findings

▶ Watch: Introducing the feedback control system analogy for PEFT (2:20)

The central and most significant finding of this research is the reinterpretation of LoRA and other PEFT algorithms not merely as weight-tuning techniques, but as implicitly designed control systems. This paradigm shift from a weight-based fine-tuning perspective to a state-based fine-tuning one forms the theoretical backbone of their contributions. The authors establish a compelling connection between the small, inserted PEFT modules and the controllers found in classical feedback control systems, where these modules respond to input and subtly alter the model's output without necessitating full model re-training.

Building upon this novel understanding, the paper proposes a specific implementation strategy: adopting a parallel control scheme. This approach involves adding new control blocks in parallel to the original neural network layers, crucially ensuring that these control blocks are decoupled and independent of the original system's formulation. This parallel design offers several key benefits, including simpler theoretical analysis—especially when compared to the highly non-linear nature of traditional LoRA which often falls into the "fully non-linear control system" category.

From a practical standpoint, the most impactful finding is the substantial memory reduction achieved through this state-based parallel control. The authors demonstrate that while LoRA weights themselves consume a relatively small fraction of total memory (e.g., 15%), the dominant memory consumption comes from storing intermediate states (XT0, XT1, etc.) required for gradient computation with respect to the LoRA weights. By shifting to state-based tuning with a parallel shortcut, their method allows for the release of these large intermediate states, leading to significant memory savings.

The experimental validation confirms these benefits:

• Significant Memory Savings: The proposed parallel control method achieves substantial memory reduction compared to both LoRA and DoRA across various configurations.
• Reduced Training Time: Alongside memory savings, the method also leads to noticeable reductions in computation time.
• Maintained or Improved Performance: Crucially, these efficiency gains do not come at the cost of model performance. The proposed method achieves results that are on par with, or even slightly better than, the original LoRA and DoRA baselines.
• Enhanced Accessibility: Thanks to the memory reduction, the researchers successfully fine-tuned large language models like Llama-2 7B and Llama-3 8B on a consumer-level NVIDIA 3090 GPU, utilizing only 20-23 gigabytes of VRAM. This makes advanced LLM fine-tuning accessible on more common and affordable hardware, broadening the scope for research and application.

In essence, the key findings revolve around a profound theoretical re-conceptualization of PEFT, leading to a practical, efficient, and performant fine-tuning methodology that addresses critical resource constraints in modern AI development.

Technical Deep Dive

▶ Watch: Shifting focus from weight-based to state-based fine-tuning (4:40)

The technical core of this work lies in re-framing Parameter-Efficient Fine-Tuning (PEFT), and specifically LoRA, through the lens of classical control theory. This re-conceptualization moves away from the conventional weight-tuning perspective towards a state-based fine-tuning approach, culminating in a parallel control architecture for improved efficiency.

Control Systems Analogy

The speakers draw a direct analogy between PEFT algorithms and feedback control systems. In a typical control system, an input signal passes through a "system" to produce an output. If this output deviates from a desired reference target, a feedback controller is introduced to steer the system. Similarly, in PEFT, the base model acts as the "system," which remains largely untouched. Small, inserted modules (like LoRA layers or adapters) function as "controllers." These modules respond to the input and subtly alter the model's intermediate states or final output, effectively "controlling" its behavior without requiring a full fine-tuning of the massive base weights.

The connection is further formalized by considering a Linear Time-Invariant (LTI) system analogy. If the dynamics of the original system are represented by XT+1 = A XT + B UT (where XT is the state and UT is the control input), and the control input UT is defined as AR XT + BR XT (where AR and BR are low-rank matrices), this formulation directly recovers the LoRA mechanism. This suggests that LoRA, while originally conceived as a weight-tuning technique, implicitly and unintentionally designs a control system.

Differences and Evolution to State-Based Tuning

While the analogy is strong, the authors highlight key differences that motivate their shift:

1. Dimensionality of States and Controls: In classical control, XT (system state) and UT (control input) do not necessarily have to be of the same shape or dimension. This implies that the control U can be an independent variable that perturbs the state, rather than merely a "completion" or "difference" to the weight tuning.
2. Non-Linear Control: The control function does not have to be linear. More complex, non-linear functions can be employed, moving beyond simple linear perturbations.
3. State-Centric Application: Crucially, control is typically applied to the states of a system, not its underlying parameters (weights). Feedback control systems usually keep the original system intact, building new "edges" or pathways that influence the flow of information or state.

These observations lead to the concept of state-based fine-tuning. Instead of viewing a neural network as a series of layers with weight matrices, it's considered a directed acyclic graph (DAG) where computation is entirely based on the flow of states. What LoRA traditionally does is modify the "flow map" between two states (e.g., Q and V states in attention blocks). However, state-based fine-tuning offers greater flexibility: one can modify any intermediate state with a control function, build new edges between arbitrary states, or even reroute the entire information flow. For example, a new edge could be added from state X1 directly to X6 if X6 requires input from X1.

Hierarchy of Control Systems and Parallel Control Proposal

The talk then reviews the hierarchy of classical control systems:

1. LTI System: Simple, elegant, and analyzable, where controls are added in parallel to a linear time-invariant system.
2. Non-linear Time-Invariant System: The original system dynamics are non-linear.
3. Control Affine System: A more widely used form where the control input is decoupled from the non-linear dynamics, typically expressed as dX/dt = f(X) + g(X)U.
4. Fully Non-linear Control System: The most complex category, where control variables are deeply intertwined with the system's non-linear dynamics.

Interestingly, the speaker notes that LoRA and many other PEFT algorithms, despite their apparent simplicity, often fall into the fully non-linear control system category due to their interaction with the base model's non-linearities (e.g., tuning Q and V blocks). While effective, this non-linear nature makes theoretical analysis extremely difficult.

To address this, the paper proposes to operate at the level of a control affine system by implementing a parallel control scheme. The core idea is to build a new control block, or "edge," that is entirely independent of the original block and operates in parallel to it. Formally, this means keeping the original formulation intact while adding a new control block in parallel. For instance, instead of modifying W_Q with W_Q + A_Q B_Q, they might add a new path X_in -> Control_Block -> X_out that operates independently and then combines its output with the original path.

Benefits of Parallel Control

The parallel nature of the control offers significant advantages:

• Simpler Theoretical Analysis: For linear neural networks, whether tuning each block individually with a low-rank matrix or perturbing all blocks with a new control matrix, the performance can be similar with sufficient rank. More profoundly, for non-linear systems where the gradient of the original function F might be singular, embedding a control system into the original system might only span a proper subspace. In contrast, putting the control outside (in parallel) allows it to span a larger, or even the whole, space, providing greater expressivity and control authority.
• Memory Reduction: This is the most crucial practical benefit. In conventional LoRA, computing gradients with respect to the low-rank weights (e.g., AT2) requires storing the forward pass intermediate states (e.g., XT1) in memory. These intermediate states, especially in deep networks, consume a significantly larger portion of memory than the LoRA weights themselves (e.g., 15% for LoRA weights vs. 85% for states). By implementing a direct shortcut from XT0 to XT2 using a new low-rank matrix for this state-based edge, the intermediate state XT1 can be released from memory. This "state-releasing" mechanism is the primary driver of the observed memory savings.

In summary, the technical proposal leverages a profound theoretical re-understanding of PEFT as control systems to engineer a parallel control affine system that directly manipulates states, thereby achieving substantial memory efficiency and analytical tractability.

Experimental Setup & Results

▶ Watch: Neural network as DAG; rerouting information flow (6:00)

To validate the theoretical insights and practical benefits of their proposed state-based fine-tuning with parallel control, the authors conducted a series of experiments on standard benchmarks using prominent large language models.

Experimental Setup

• Models Evaluated: The experiments focused on two widely-used large language models: Llama-2 7B and Llama-3 8B. These models represent significant computational challenges for fine-tuning, making them ideal candidates to demonstrate memory efficiency.
• Benchmarks: The evaluation was performed on a commonsense benchmark, implying tasks that test a model's understanding and reasoning abilities in everyday scenarios. While specific dataset names are not detailed in the transcript, the focus is on general commonsense reasoning.
• Baselines: The proposed method was rigorously compared against established PEFT techniques, specifically LoRA and DoRA. This ensures a fair comparison against state-of-the-art parameter-efficient approaches.
• Tuning Scope: For comprehensive evaluation, the fine-tuning was applied to all five key blocks within the transformer architecture: the Query (Q), Key (K), and Value (V) projection matrices in the attention mechanism, as well as the Up-projection (U) and Down-projection (D) matrices in the MLP layers. This covers a significant portion of the trainable parameters in PEFT methods.
• Proposed Solutions (Control Configurations): Two main configurations of their parallel control approach were tested:

1. Hybrid Approach: Using parallel control specifically for the U and D blocks within the MLP layers (where intermediate states are typically very large), while retaining traditional LoRA for the Q, K, and V blocks.
2. Full Control Approach: Applying parallel control for all five blocks (Q, K, V, U, D). This represents the full adoption of their state-based parallel control paradigm.

Headline Results

The experimental results unequivocally demonstrate the superiority of the proposed parallel control method across key metrics:

• Memory Savings: The most significant headline result is the "significant" memory saving achieved by the parallel control solutions compared to both LoRA and DoRA. The talk explicitly highlights that while LoRA weights themselves are small (e.g., consuming only 15% of total memory in an example), the storage of intermediate states (XT0, XT1, etc.) for gradient computation dominates memory usage. By enabling the release of these intermediate states, the parallel control method drastically reduces the overall memory footprint.
• Computation Time Reduction: Beyond memory, the proposed method also achieved savings in computation time for both Llama-2 and Llama-3 models. This indicates that the architectural changes not only reduce VRAM requirements but also streamline the training process, leading to faster iteration cycles.
• Performance Parity (or Improvement): Crucially, these efficiency gains do not come at the expense of model performance. The results show that the performance of the parallel control method is "on par with the original one or even slightly better" when compared to LoRA and DoRA. This confirms that the re-conceptualization and architectural changes are not merely optimizations but potentially enhancements to the fine-tuning process.
• Accessibility for Consumer Hardware: A highly impactful practical result is the ability to fine-tune Llama-2 7B and Llama-3 8B models on a consumer-level NVIDIA 3090 GPU. The talk specifies that this was achieved using only 20 to 23 gigabytes of VRAM. This is a critical breakthrough, as these models typically require professional-grade GPUs with much larger memory capacities. The speaker noted that while training on a 3090, the performance "only decreased slightly" compared to larger setups, likely due to necessary reductions in batch size to fit the memory constraints. This still represents a massive leap in accessibility.

In summary, the experiments provide strong empirical evidence that the state-based fine-tuning with parallel control offers a superior alternative to traditional weight-based PEFT methods, delivering substantial resource efficiencies while maintaining or even improving model performance, thereby making advanced LLM fine-tuning more accessible.

Practical Implications

▶ Watch: Paper's proposal: Focusing on control affine systems (7:59)

The research presented in "From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control" carries profound practical implications for various stakeholders in the AI/ML ecosystem, from individual practitioners to large infrastructure teams.

For practitioners and model builders, the most immediate and impactful benefit is the democratization of large language model (LLM) fine-tuning. The ability to fine-tune models like Llama-2 7B and Llama-3 8B on a single consumer-level NVIDIA 3090 GPU with only 20-23 GB of VRAM is a game-changer. Traditionally, such tasks demanded expensive, enterprise-grade hardware (e.g., A100s or H100s) with 40GB or 80GB of VRAM, often in multi-GPU setups. This memory reduction lowers the barrier to entry significantly, allowing more researchers, startups, and hobbyists to experiment with and adapt powerful LLMs to their specific needs without prohibitive hardware investments. This could accelerate innovation and deployment of specialized AI models across a wider range of applications.

For infrastructure teams and those responsible for model deployment, the implications are equally significant. The substantial reduction in memory consumption and training time translates directly into lower operational costs and more efficient resource utilization. Less VRAM per training job means more models can be trained concurrently on existing hardware, or fewer GPUs are needed for the same workload, leading to reduced electricity consumption and hardware expenditures. Faster training times mean quicker iteration cycles, allowing teams to develop and deploy updated models more rapidly in response to evolving requirements or data distributions. This efficiency can be particularly critical in cloud environments where compute and memory are billed resources.

The shift from weight-based to state-based fine-tuning also offers a new conceptual framework for model developers. Understanding PEFT methods as control systems opens up new avenues for designing even more efficient and effective adaptation strategies. Instead of being constrained by modifying specific weight matrices, developers can now think about manipulating intermediate states, injecting new pathways, or rerouting information flow within the neural network graph. This expanded design space could lead to the discovery of novel PEFT techniques tailored for specific architectures or tasks, potentially leading to even greater efficiencies or performance gains.

Tradeoffs and Limitations: While the benefits are substantial, it's important to consider potential tradeoffs. The speaker noted a "slight" performance decrease when training on the NVIDIA 3090, which was attributed to the necessity of shrinking the batch size to fit within the limited VRAM. This highlights a common dilemma in resource-constrained environments: memory limitations often force smaller batch sizes, which can sometimes (though not always significantly) impact final model quality or convergence speed. Practitioners must weigh this potential slight performance reduction against the substantial cost savings and accessibility gains.

Furthermore, while the theoretical shift provides a simpler framework for analysis (especially moving from fully non-linear control to control affine systems), it also requires a conceptual adjustment for engineers accustomed to the weight-tuning paradigm. The practical implementation of state-based control might introduce new complexities or considerations that need to be carefully managed. The talk primarily focuses on memory and time, and while performance is maintained, detailed analysis of how this state-based perturbation affects model robustness, generalization to out-of-distribution data, or transferability across tasks is not explicitly covered. Nevertheless, the overall practical implications are overwhelmingly positive, ushering in an era of more accessible and resource-efficient large model fine-tuning.

Key Takeaways

• PEFT Reconceptualization: LoRA and other Parameter-Efficient Fine-Tuning (PEFT) algorithms can be fundamentally reinterpreted as control systems rather than solely weight-tuning techniques.
• Shift to State-Based Fine-Tuning: The focus of fine-tuning can effectively shift from modifying model weights to perturbing and controlling the intermediate states within the neural network's computational graph.
• Parallel Control for Efficiency: Implementing a parallel control scheme, particularly at the control affine system level, offers simpler theoretical analysis and enables significant memory reduction by allowing the release of large intermediate states.
• Significant Resource Savings: The proposed method achieves substantial memory savings and reduced training time compared to conventional LoRA and DoRA, while maintaining or even slightly improving model performance.
• Enhanced Accessibility: This approach makes fine-tuning large language models like Llama-2 7B and Llama-3 8B feasible on consumer-grade hardware, specifically NVIDIA 3090 GPUs with only 20-23 GB of VRAM.
• New Design Space: The control theory perspective opens new avenues for designing more flexible and efficient PEFT strategies, allowing for the creation of new computational pathways and arbitrary state manipulations.

About the Speaker(s)

The talk was presented by Chi Zhang, and is a joint work with REN Lianhai, Jingpu Cheng, and Qianxiao Li. All four speakers are identified as a group of scientists affiliated with the National University of Singapore. Their collective research focuses on advancing the understanding and efficiency of machine learning algorithms, particularly in the domain of parameter-efficient fine-tuning for large models.