LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

Speakers: Yuanhe Zhang; Fanghui Liu; Yudong Chen

Conference: ICML 2025

YouTube: https://slideslive.com/39044099

Overview

This talk, presented by Fanghui Liu at ICML 2025, introduces LoRA-One, a novel approach to fine-tuning Large Language Models (LLMs) that redefines the trade-off between performance and efficiency. The core premise of LoRA-One is that a single computation of the full gradient, combined with a theory-grounded initialization strategy, can be sufficient for effectively fine-tuning LLMs using Low-Rank Adaptation (LoRA). The work is a collaboration with Yuanhe Zhang (lead PhD student) and Yudong Chen.

The presentation delves into the theoretical underpinnings of LoRA's training dynamics, revealing that the learned low-rank matrices (A and B) align with specific singular subspaces of the one-step full gradient. By leveraging this insight, LoRA-One proposes a spectral initialization technique that bypasses the need for extensive iterative training steps. This leads to substantial efficiency gains, reducing fine-tuning times from tens of seconds to less than one second for certain tasks, while maintaining or even improving performance. The implications are significant for practitioners seeking to accelerate LLM development and deployment, particularly in resource-constrained environments or for rapid experimentation cycles.

Background

▶ Watch: Introduction: Fine-tuning LLMs, LoRA, performance vs efficiency (0:00)

The landscape of Large Language Model (LLM) adaptation is broadly characterized by two main strategies: full fine-tuning and Parameter-Efficient Fine-Tuning (PEFT). Full fine-tuning involves updating all parameters of a pre-trained LLM on a downstream task, typically leading to high performance but at a considerable computational and memory cost. This approach is often prohibitive for models with billions of parameters, requiring extensive GPU resources and long training times.

In response, PEFT methods have emerged to mitigate these challenges by only updating a small subset of parameters or introducing a few trainable parameters. Among these, LoRA (Low-Rank Adaptation) has gained widespread popularity. LoRA operates on the principle that the "feature shift" or weight update required for fine-tuning a pre-trained model can be represented by a low-rank matrix. Instead of directly fine-tuning the full weight matrix W, LoRA approximates its update ΔW as a product of two smaller, low-rank matrices, A and B (i.e., ΔW = AB). During fine-tuning, only A and B are trained, significantly reducing the number of trainable parameters and computational overhead. Typically, A is initialized with random Gaussian values, and B is initialized to zero, or both are randomly initialized.

Despite its efficiency benefits compared to full fine-tuning, standard LoRA still involves iterative gradient descent or AdamW optimization over multiple epochs. The speakers argue that this iterative process, particularly its convergence rate and final performance, is often suboptimal due to the initial random initialization of the low-rank matrices. The core problem addressed by this work is understanding how A and B evolve during training and whether a more principled initialization can accelerate convergence and improve performance without sacrificing the efficiency gains inherent to LoRA. The research aims to understand this from two perspectives: speeding up efficiency without performance decrease, and improving performance under the same computational budget.

Key Findings

▶ Watch: One-step full gradient drives LoRA's dynamics (2:40)

The talk presents several pivotal findings that challenge conventional wisdom in LoRA fine-tuning and introduce a more efficient paradigm:

1. Subspace Alignment Theory: Through theoretical analysis, initially demonstrated for linear models and then extended to non-linear models (though with greater analytical complexity), the authors prove that the trained LoRA matrices, specifically B_t and eventually A_t, align to specific singular subspaces of the one-step full gradient (G). B_t aligns to the top r_star right-hand singular subspace of G with a zero angle, while A_t aligns to the top r_star left-hand singular subspace of G after log d steps (given random Gaussian initialization). This alignment indicates that the essential information for optimal fine-tuning is encoded within this single full gradient computation.
2. LoRA-One Algorithm (Spectral Initialization): Based on the subspace alignment theory, the authors propose LoRA-One, an algorithm that uses spectral initialization. Instead of random initialization and iterative training, LoRA-One directly computes the one-step full gradient G (the gradient of the full fine-tuning loss with respect to the pre-trained weights) and then performs a Singular Value Decomposition (SVD) on G. The top r singular vectors are then used to initialize A_0 and B_0. This "gold standard" initialization strategy ensures that the starting point is already very close to the optimal solution, significantly accelerating convergence.
3. Dramatic Efficiency Improvements: For small-scale datasets (e.g., CoLA, MRPC with less than 10k samples), LoRA-One, requiring only the computation of the one-step full gradient and its SVD, can match the performance of a conventionally trained LoRA model (run for one epoch). Critically, this is achieved with a massive reduction in computational time: LoRA-One takes less than one second, whereas standard LoRA fine-tuning can take 20-40 seconds or more. This represents a 20x to 40x speedup without performance degradation.
4. Performance Enhancement under Fixed Compute: For larger models and datasets (e.g., fine-tuning Llama 2 7B over several epochs), LoRA-One demonstrates the ability to significantly improve LoRA's performance while maintaining the same time and memory cost. This shows that the superior initialization translates not only to faster convergence but also to potentially better final model quality within a given computational budget.
5. Clarification of Prior Work: The theory also clarifies misconceptions in previous research, specifically criticizing LoRA-GA. LoRA-GA assumed that the gradients of A and B provide 2r degrees of freedom for approximating the full gradient, leading to an initialization strategy that uses higher-rank singular subspaces. LoRA-One's theory proves that the gradients of A and B are not independent and do not provide 2r degrees of freedom in the assumed manner, explaining why LoRA-GA can get stuck or perform suboptimally by aligning to the "wrong" singular subspace.

Technical Deep Dive

▶ Watch: Proof: LoRA's Bt aligns to one-step gradient (3:50)

The technical core of LoRA-One lies in understanding the dynamics of LoRA training and leveraging this understanding for a superior initialization strategy.

LoRA's Formulation and Problem Statement:

In LoRA, the update to a pre-trained weight matrix W is approximated as ΔW = B A, where A ∈ ℝ^(d_in × r) and B ∈ ℝ^(d_out × r). Here, r is the chosen low-rank, which is typically much smaller than min(d_in, d_out). The speaker explicitly states their assumption that r (the rank used for LoRA) is larger than r_star (the intrinsic rank of the true feature shift ΔW). The goal of LoRA training is to estimate the unknown target feature shift ΔW by learning A and B such that BA approximates ΔW.

The Dynamics of LoRA Training:

The speaker highlights that even for a simplified linear model where pre-training and fine-tuning are linear, the dynamics of A_t and B_t (the matrices at iteration t) are inherently nonlinear. The recursive relationship for A_t and B_t contains a nonlinear term. If this term were absent, the system would be a classical linear dynamic system with a closed-form solution.

The crucial observation is that the dynamics of A_t and B_t are driven by the one-step full gradient (G). This G is defined as the gradient of the full fine-tuning loss function (e.g., square loss in the linear model setting) with respect to the initial full model weights, computed after the very first step of full fine-tuning. This G encapsulates the information about the unknown feature shift ΔW that the model needs to learn.

Subspace Alignment Theory:

The central theoretical contribution is the proof of subspace alignment. The authors use Singular Value Decomposition (SVD), where a matrix M can be decomposed into U S Vᵀ, with U being the left-hand singular subspace and V being the right-hand singular subspace.

The theory states:

1. Alignment of B_t: For the linear model, after gradient descent or AdamW training, the matrix B_t will align to the top r_star right-hand singular subspace of the one-step full gradient G. Critically, the angle between these two subspaces is proven to be zero. This means B_t perfectly captures the most important directions from the full gradient.
2. Alignment of A_t: The alignment of A_t is slightly more complex due to its typical random Gaussian initialization (as opposed to B_t's zero initialization). However, it is proven that A_t will align to the top r_star left-hand singular subspace of G after log d steps, with the angle between the subspaces being small with high probability.

These theoretical findings are supported by numerical results on T5 models, showing that the angle between the learned subspaces and the target singular subspaces of G indeed decreases during fine-tuning, starting from around 1 and converging to values like 0.2 or 0.4.

LoRA-One: Spectral Initialization:

The subspace alignment theory provides the blueprint for LoRA-One. If the trained A and B eventually align to the singular subspaces of G, why not initialize them directly from G?

The spectral initialization strategy is as follows:

1. Compute the one-step full gradient G. This involves computing the gradient of the fine-tuning loss with respect to the entire pre-trained model parameters W_0 just once.
2. Perform SVD on G: G = U S Vᵀ.
3. Initialize A_0 and B_0 using the top r singular vectors from U and V. Specifically, B_0 is initialized using the top r right-singular vectors (columns of V), and A_0 is initialized using the top r left-singular vectors (columns of U). The speaker mentions A_0 B_0 directly recovering ΔW, implying a specific scaling with singular values, though the exact formulation isn't detailed in the transcript.

This direct initialization ensures that A_0 and B_0 are already aligned to the optimal subspaces from the very beginning. This makes LoRA-One's initialization a "gold standard" or "best initialization strategy" because it starts the optimization process much closer to the global minimum, as visualized by phase portraits in dynamic systems. This closeness drastically accelerates convergence and can improve the quality of the final solution.

Critique of LoRA-GA:

The theoretical framework also allows for a critical re-evaluation of prior work. The speaker specifically points to LoRA-GA, a method that aimed to align LoRA's gradient with the full fine-tuning gradient. LoRA-GA incorrectly assumed that the gradients of A and B could provide 2r independent degrees of freedom for approximating the optimal ΔW. Based on this, it chose singular vectors from r+1 to 2r for initialization. However, LoRA-One's theory demonstrates that the gradients of A and B are not independent in the way LoRA-GA assumed, meaning there aren't 2r independent degrees of freedom. Consequently, LoRA-GA's initialization leads to alignment with the "wrong" singular subspace, causing the loss to get stuck and performance to be suboptimal, which is consistent with the experimental observations.

Experimental Setup & Results

▶ Watch: LoRA-One's efficiency: Matches LoRA performance significantly faster (6:00)

The research backs its theoretical claims with practical experiments on various LLMs and datasets, showcasing both efficiency and performance benefits.

Models and Baselines:

• The primary models used for evaluation include T5 for initial verification and smaller-scale NLP tasks, and Llama 2 7B for demonstrating capabilities on larger, more representative LLMs.
• The main baseline for comparison is standard LoRA, typically trained for one epoch or several epochs, depending on the experiment's goal.

Datasets:

• For demonstrating efficiency gains without performance decrease, the experiments utilized small-scale NLP datasets such as CoLA (Corpus of Linguistic Acceptability) and MRPC (Microsoft Research Paraphrase Corpus). These datasets are characterized by having less than 10,000 samples.
• For demonstrating performance improvement under the same computational budget, the experiments likely involved larger datasets or more complex tasks, though specific names beyond "NLP data set" are not mentioned for the Llama 2 experiments.

Metrics:

• Performance: While not explicitly named, for NLP tasks, this would typically involve metrics like accuracy, F1-score, or other task-specific evaluation scores. The talk implicitly refers to "matching performance" or "improving performance."
• Efficiency: Measured in terms of time cost (seconds) and memory cost.

Headline Results:

1. Efficiency for Small-Scale Datasets:

• On small NLP datasets like CoLA and MRPC, LoRA-One (using spectral initialization and effectively one "step" of the full gradient computation) was able to match the performance of standard LoRA trained for a full epoch.
• The time difference was dramatic: LoRA-One took less than one second, while standard LoRA required 20 to 40 seconds or more. This represents a 20x to 40x speedup, demonstrating that for these tasks, a single full gradient computation is indeed sufficient when coupled with the correct initialization.

1. Performance Improvement for Larger Models/Datasets:

• When fine-tuning Llama 2 7B (a significantly larger model) over several epochs, LoRA-One demonstrated the ability to significantly improve LoRA's performance.
• Crucially, this performance boost was achieved while maintaining "almost the same time cost and memory cost" as standard LoRA. This highlights LoRA-One's ability to extract more performance from the same computational budget, suggesting a more efficient trajectory through the optimization landscape.

1. Numerical Verification of Subspace Alignment:

• Experiments fine-tuning the T5 model showed that the angle between the learned A_t/B_t subspaces and the target singular subspaces of the one-step full gradient G indeed decreased from around 1 (at initialization) to 0.2-0.4 during fine-tuning. This empirically validates the theoretical prediction of subspace alignment.

1. Critique of LoRA-GA (Experimental Evidence):

• While not detailed with specific numbers, the speaker notes that LoRA-GA's performance "will get stuck" regarding the loss. This experimental observation supports the theoretical critique that LoRA-GA's incorrect assumptions about gradient independence lead to suboptimal optimization paths.

In summary, the experimental results provide strong empirical evidence for LoRA-One's claims, demonstrating both remarkable efficiency gains for smaller tasks and tangible performance improvements for larger models under equivalent computational constraints.

Practical Implications

▶ Watch: LoRA-One's performance: Improves LoRA under same compute (6:30)

The development of LoRA-One carries substantial practical implications for various stakeholders involved in the LLM lifecycle, from model builders to infrastructure teams.

For Practitioners and Model Builders:

• Accelerated Iteration Cycles: The most direct benefit is the dramatic reduction in fine-tuning time for many tasks. For small-to-medium scale datasets, achieving competitive performance in under a second (compared to tens of seconds) means practitioners can iterate on task-specific fine-tuning experiments much faster. This rapid feedback loop enables quicker hypothesis testing, model selection, and hyperparameter tuning.
• Improved Model Quality (Fixed Budget): For larger models and more complex tasks where multiple epochs are still necessary, LoRA-One provides a superior starting point. This "gold standard" initialization means that for the same computational budget (e.g., same number of epochs, same training time), LoRA-One can yield a better-performing model than standard LoRA. This is critical for achieving state-of-the-art results efficiently.
• More Principled Fine-Tuning: Moving beyond heuristic random initialization, LoRA-One offers a theoretically grounded method. This reduces the "black magic" often associated with finding good initializations and provides a more predictable path to convergence.

For Infrastructure Teams and Deployers:

• Reduced GPU Resource Consumption: The ability to achieve competitive performance with significantly fewer training steps (or even effectively one step for small datasets) translates directly to lower GPU utilization. This can free up valuable and expensive GPU resources, allowing for more concurrent experiments, serving more users, or reducing operational costs.
• Lower Barrier to Entry: For teams with limited computational resources, LoRA-One makes fine-tuning large models more accessible. The reduced training time and potential for better performance on less compute democratizes access to advanced LLM capabilities.
• Efficient Model Updates: In deployment scenarios, models often need to be continuously updated with new data. LoRA-One could facilitate faster and more efficient model updates, ensuring models remain relevant and accurate without prolonged downtime or significant re-training costs.

Tradeoffs and Limitations:

• "One-Step Sufficiency" is Contextual: While the headline "one-step full gradient could suffice" is powerful, the talk clarifies that this is particularly true for "small-scale data sets" where LoRA-One can match LoRA's one-epoch performance. For larger datasets, while the initialization is still superior and improves performance, some iterative training (multiple epochs) is still required to close the performance gap entirely or achieve optimal results. The "one step" refers to the calculation of the full gradient for initialization, not necessarily the entire fine-tuning process for all scenarios.
• Computational Cost of Full Gradient: The initial step involves computing the full gradient of the entire model, which can be memory-intensive, especially for very large models. However, this is a one-time cost and is offset by the subsequent efficiency gains and reduced iterative training.
• Complexity of Non-Linear Analysis: The theoretical proofs for non-linear models are "quite complex" and were skipped in the talk. While numerical results support the theory, a deeper analytical understanding for complex non-linear architectures could provide further insights and guarantees.
• Generalizability of r > r_star: The assumption that the chosen LoRA rank r is larger than the true intrinsic rank r_star of the feature shift is crucial. While often a reasonable assumption in practice, understanding its boundaries and impact on LoRA-One's effectiveness is important.

Overall, LoRA-One offers a powerful, theoretically-backed paradigm shift for LoRA fine-tuning, promising significant efficiency gains and performance improvements that can streamline LLM development and deployment across various applications.

Key Takeaways

• LoRA's Suboptimal Initialization: Traditional LoRA fine-tuning, despite its efficiency, often suffers from suboptimal random initialization, leading to slower convergence and potentially lower performance.
• The Power of One-Step Full Gradient: The essential information for optimal LoRA fine-tuning is encoded within the one-step full gradient, which captures the target feature shift ΔW.
• LoRA-One's Spectral Initialization: The proposed LoRA-One algorithm leverages Singular Value Decomposition (SVD) of this one-step full gradient to perform spectral initialization, directly aligning the LoRA matrices (A and B) to the optimal singular subspaces from the start.
• Dramatic Efficiency Gains: For small-scale datasets (e.g., CoLA, MRPC), LoRA-One can match the performance of standard LoRA (trained for one epoch) in under one second, representing a 20x to 40x speedup.
• Improved Performance with Fixed Compute: For larger models like Llama 2 7B, LoRA-One significantly improves performance over standard LoRA while maintaining the same computational budget (time and memory).
• Theoretical Clarity and Correction: The underlying theory clarifies misconceptions in prior work (e.g., LoRA-GA's incorrect assumptions about gradient independence), providing a more accurate understanding of LoRA's dynamics.

About the Speaker(s)

The talk was presented by Fanghui Liu, who identified himself as a "learning theory guy" and collaborator on this research. He is actively involved in deep learning theory, particularly concerning global convergence for non-linear models. The primary author and PhD student on this work is Yuanhe Zhang, who unfortunately could not attend the conference due to visa issues. Yudong Chen from Madison was also a collaborator on this research. The team's work focuses on bridging theoretical understanding with practical efficiency in large language models.