LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail)

LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail)

Speakers: Junsu Kim, Jaeyeon Kim, Ernest Ryu

Conference: ICML 2025

YouTube: https://slideslive.com/39044098

Overview

Low-Rank Adaptation (LoRA) has emerged as a cornerstone technique for parameter-efficient fine-tuning of large pre-trained models. By introducing low-rank updates to specific layers, LoRA significantly reduces the number of trainable parameters, enabling more efficient adaptation to downstream tasks without sacrificing performance. This talk, presented by Ernest Ryu at ICML 2025, delves into the theoretical underpinnings of LoRA, offering a rigorous analysis of its convergence properties. The central thesis is encapsulated in its provocative title: LoRA training provably converges to a low-rank global minimum, or it fails loudly, though the latter is argued to be highly improbable in practice.

The research, a collaborative effort with Junsu Kim and Jaeyeon Kim, addresses a critical gap in the understanding of LoRA's effectiveness. While LoRA is widely adopted, prior theoretical work often relied on strong, sometimes unrealistic, assumptions such as the Neural Tangent Kernel (NTK) linearization or limitations on dataset size. This work introduces a more relaxed set of assumptions, providing a robust theoretical framework that aligns more closely with practical LoRA fine-tuning scenarios. It demonstrates that under reasonable conditions, LoRA's optimization landscape is remarkably well-behaved, ensuring convergence to desirable solutions.

This article explores the intricate details of this theoretical breakthrough, beginning with the foundational concepts of LoRA and optimization theory, then moving into the novel assumptions and the main theorem's implications. It unpacks the role of implicit biases like zero initialization and weight decay in guiding LoRA towards optimal, low-rank solutions and discusses the practical ramifications for practitioners and researchers. By demystifying LoRA's convergence behavior, this work bolsters confidence in its continued use and offers insights for future advancements in efficient model adaptation.

Background

▶ Watch: Introduction to LoRA: mechanism, initialization, and application (0:00)

To appreciate the significance of this work, it's essential to understand the context of LoRA and the theoretical challenges it presents. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. Instead of fine-tuning all parameters of a large pre-trained model, LoRA introduces a low-rank update to existing linear layers. Specifically, for a pre-trained weight matrix $X_0$, LoRA replaces it with $X_0 + A B^\top$, where $A$ is a tall-skinny matrix (e.g., $d \times R$) and $B$ is a fat-wide matrix (e.g., $d \times R$). The product $A B^\top$ yields a rank-$R$ update, where $R$ is typically a small value like 16. During LoRA training, the original pre-trained weights $X_0$ are frozen. Customarily, $A$ is initialized with i.i.d. Gaussians, and $B$ is initialized to all zeros, ensuring that the initial update $A B^\top$ is zero. In transformer architectures, LoRA adapters are commonly placed on the linear layers responsible for producing Query (Q), Key (K), and Value (V) vectors, as well as in the position-wise Feed-Forward Network (FFN) layers. Full fine-tuning would optimize $X_0 + X_{\text{square}}$, where $X_{\text{square}}$ is the full-rank update. LoRA, however, restricts $X_{\text{square}}$ to $A B^\top$ and optimizes only $A$ and $B$.

A crucial aspect of LoRA training, often overlooked in theoretical analyses, is the use of weight decay. When applied to LoRA, weight decay is typically applied separately to the factors $A$ and $B$. This translates to a regularization term in the loss function, such as $\lambda_A ||A||_F^2 + \lambda_B ||B||_F^2$, where $||\cdot||_F$ denotes the Frobenius norm. A key insight highlighted by the speakers is that this separate Frobenius norm regularization on $A$ and $B$ is mathematically equivalent to applying a nuclear norm regularization on the product update $X_{\text{square}} = A B^\top$. The nuclear norm, defined as the sum of singular values of a matrix, is well-established in signal processing and machine learning for its property of inducing low-rank solutions. Therefore, weight decay on LoRA factors implicitly biases the training towards low-rank updates.

Prior theoretical work analyzing LoRA often relied on the Neural Tangent Kernel (NTK) assumption. This assumption posits that the first-order Taylor expansion of the loss function with respect to parameter updates holds sufficiently well throughout training. In the context of fine-tuning, where parameter updates are expected to be small, this assumption appears plausible. Indeed, empirical work has shown the NTK assumption to be reasonably accurate in certain prompt-based fine-tuning settings. However, it has also been observed that the NTK assumption does not hold universally across all practical LoRA fine-tuning scenarios, limiting the generalizability of theories built upon it.

To understand the convergence guarantees, it's also necessary to define key optimization concepts. A point $U$ is a first-order stationary point if the gradient of the loss function at that point is zero ($\nabla L(U) = 0$). It is a second-order stationary point if, in addition to having a zero gradient, its Hessian matrix is positive semi-definite. A strict saddle point is a first-order stationary point that is not a second-order stationary point, meaning the Hessian has at least one negative eigenvalue. In optimization, particularly with Stochastic Gradient Descent (SGD), there's a well-known result by Ge et al. and Lee et al. stating that SGD, with probability one, will not converge to strict saddle points; it will only converge to second-order stationary points. This is a powerful result, but it doesn't guarantee convergence to a good solution, as a second-order stationary point could still be a spurious local minimum. The challenge is to prove that, in the context of LoRA, all second-order stationary points are indeed desirable global minima.

A prior theoretical result in LoRA analysis states that if the LoRA rank $R$ is larger than $\sqrt{N}$ (where $N$ is the fine-tuning data size) and if a linearized neural network (implying the NTK assumption) is considered with a small random perturbation, then all second-order stationary points are global minimizers. This theorem suggested that LoRA training has no spurious local minima under these conditions. While insightful, this prior work was constrained by assumptions such as a relatively small fine-tuning dataset size (e.g., $N \approx 1000$) and the pervasive NTK linearization assumption, which this new work aims to relax.

Key Findings

▶ Watch: NTK assumption: its role and limitations in LoRA analysis (2:55)

The current work significantly advances the theoretical understanding of LoRA by relaxing several strong assumptions prevalent in prior research, particularly the reliance on the NTK assumption and limitations on dataset size. This allows for a more general and practically relevant analysis.

The cornerstone of this new theory rests on a few critical assumptions:

1. Existence of a Low-Rank Minimizer: The primary assumption is that a global minimizer, denoted as $X_{\text{star}}$, exists for the full fine-tuning loss (without any linearization). Crucially, this $X_{\text{star}}$ is assumed to possess a low rank, $R_{\text{star}}$. This assumption bypasses the need for the NTK linearization, directly addressing the full, non-linear fine-tuning objective. While the reason why many fine-tuning tasks inherently admit low-rank updates remains an open question, this assumption is observed to hold approximately in practice.
2. Sufficient LoRA Rank: The LoRA module's rank $R$ must be sufficiently large to capture this true low-rank minimizer, specifically $R > R_{\text{star}}$. This ensures that the model has the capacity to reach the optimal low-rank solution. This is a strong but generally met assumption in practical settings where LoRA ranks like 16 or 32 are chosen.
3. Restricted Strong Convexity (RSC) and Restricted Strong Smoothness (RSS): Instead of global strong convexity or smoothness, which are typically not satisfied by deep learning objectives, this work relies on these "restricted" versions. These assumptions are weaker, requiring the properties to hold only locally and for deviations of small rank. The speaker claims these are more reasonable and weaker than the NTK assumption, making the analysis more robust.

Under these refined assumptions, the main theorem presents a powerful dichotomy for LoRA's convergence:

• Scenario 1: Convergence to a Good Global Minimum. If a second-order stationary point is reached, it will either be a low-rank global minimum. This is the desired outcome, where LoRA successfully finds an optimal, parameter-efficient solution.
• Scenario 2: Failure Loudly. The alternative is that the second-order stationary point has a large rank and a large magnitude. Such a solution would represent a spurious local minimum, far from the true global minimum, and would exhibit poor generalization performance. The term "failing loudly" signifies that this undesirable outcome would be immediately evident due to the high rank and large magnitude of the learned update, making it easily detectable. This is preferable to "failing silently" where a bad solution might be low-rank and small-magnitude, masquerading as a good one.

Crucially, the talk argues that LoRA training is "probably very much unlikely to fail" in the "loudly" manner, due to implicit biases inherent in its standard setup:

• Zero Initialization: LoRA's practice of initializing the $B$ factor to all zeros means the update $A B^\top$ starts at zero. This biases the optimization towards minima that are close to the initialization, i.e., those with smaller magnitude updates.
• Weight Decay: The application of weight decay (which, as discussed in the background, induces nuclear norm regularization) implicitly biases the optimization towards low-rank matrices.

Together, these implicit biases guide the optimization process towards the region of the loss landscape where only the true, low-rank, small-magnitude global minima reside, making convergence to the "loud failure" scenario highly improbable.

The theoretical findings are supported by empirical validation:

• Assumption Verification: Experiments confirm that the restricted strong convexity and restricted strong smoothness assumptions indeed hold in practice. Interestingly, these coefficients become more favorable as the LoRA rank parameter $R$ decreases, suggesting that a smaller $R$ not only offers memory benefits but also a "nicer" (more convex-like) loss landscape.
• Two Scenarios Validation: The core theorem's two scenarios were empirically observed. With the usual LoRA initialization (random $A$, zero $B$), LoRA consistently converged to the global minimum. However, when using a random non-zero initialization, the training converged to the spurious local minimum (the "bad" one with large rank and magnitude). This experiment provides concrete evidence that these undesirable spurious minima do exist but are effectively avoided by LoRA's standard initialization strategy.

In summary, this work provides strong theoretical guarantees for LoRA's convergence, under more realistic assumptions, and explains why it works so well in practice by highlighting the role of implicit biases in navigating a complex loss landscape.

Technical Deep Dive

▶ Watch: Optimization: SGD avoids strict saddles, converges to 2nd-order (4:00)

The technical exposition of this work centers on reformulating the LoRA optimization problem and analyzing its landscape under novel, relaxed assumptions.

Let the pre-trained weights of a linear layer be $X_0 \in \mathbb{R}^{d \times d}$. In full fine-tuning, one optimizes for a full-rank update $X_{\text{square}} \in \mathbb{R}^{d \times d}$, solving $\min_{X_{\text{square}}} L(X_0 + X_{\text{square}})$, where $L$ is the loss function. LoRA restricts this update to a low-rank form: $X_{\text{square}} = A B^\top$, where $A \in \mathbb{R}^{d \times R}$ and $B \in \mathbb{R}^{d \times R}$. The optimization then becomes $\min_{A, B} L(X_0 + A B^\top)$. The rank $R$ is a hyperparameter, typically much smaller than $d$, reflecting the low-rank nature of the adaptation.

A critical component of LoRA training is weight decay. When applied to $A$ and $B$ separately, the regularization term added to the loss is $\lambda_A ||A||_F^2 + \lambda_B ||B||_F^2$. The key technical insight here is the equivalence between this Frobenius norm regularization on the factors and nuclear norm regularization on the product. The nuclear norm of a matrix $M$, denoted $||M||_*$, is the sum of its singular values. It is a convex relaxation of the rank function and is known to promote low-rank solutions. This equivalence, established in matrix completion and sensing literature, means that standard LoRA weight decay implicitly encourages the learned update $X_{\text{square}}$ to be low-rank.

Prior theoretical work often employed the NTK assumption, which linearizes the neural network around its initialization. Formally, this means approximating the loss $L(X_0 + \Delta X)$ with its first-order Taylor expansion $L(X_0) + \langle \nabla L(X_0), \Delta X \rangle$. While simplifying analysis, this approximation's validity over the entire training trajectory for complex deep learning models is questionable, especially for fine-tuning tasks where the update $\Delta X$ might not remain infinitesimally small. This work explicitly avoids this strong assumption.

Instead, the core of the new analysis relies on two main assumptions regarding the loss landscape:

1. Existence of a Low-Rank Global Minimizer: It is assumed that there exists a global minimizer $X_{\text{star}}$ of the unrestricted fine-tuning loss $L(X_0 + X_{\text{square}})$ such that $X_{\text{star}}$ has a rank $R_{\text{star}}$. This $R_{\text{star}}$ is the intrinsic rank of the optimal update. This is a fundamental departure from NTK-based analyses, which often assume a linear model or rely on the fine-tuning update being small enough that $X_0$ is already close to optimal.
2. Sufficient LoRA Rank: The rank $R$ chosen for the LoRA adapters must be greater than or equal to the intrinsic rank of the true minimizer: $R \ge R_{\text{star}}$. This ensures that the LoRA parameterization has sufficient capacity to represent the optimal low-rank update. In practice, $R$ is typically chosen as 8, 16, or 32, which is often sufficient for many fine-tuning tasks.

Beyond these, the analysis employs Restricted Strong Convexity (RSC) and Restricted Strong Smoothness (RSS). These are weaker and more realistic assumptions for non-convex deep learning objectives compared to global strong convexity/smoothness.

• Restricted Strong Convexity: Roughly, an objective $f$ is RSC if, for any matrix $M$ within a certain restricted set (e.g., matrices of small rank or small magnitude), $f(X_0+M) - f(X_0)$ is lower bounded by a quadratic function of $M$.
• Restricted Strong Smoothness: Similarly, $f$ is RSS if its gradient does not change too rapidly for matrices $M$ in the same restricted set, upper bounded by a quadratic function.

These restrictions are crucial because they allow the theory to focus on the behavior of the loss function in regions relevant to low-rank updates, rather than requiring global properties that deep learning models typically lack. The speaker notes that these assumptions only need to hold locally and for deviations of small rank, making them much more plausible than the global linearization implied by NTK.

The proof methodology leverages "machinery from low-rank matrix sensing theory." This field focuses on recovering low-rank matrices from incomplete or noisy observations, often employing nuclear norm regularization and analyzing non-convex optimization landscapes. A specific technical challenge highlighted by the speaker was adapting this machinery to asymmetric LoRA factors ($A$ and $B$), as much of the existing matrix sensing literature deals with symmetric matrix factorization. This adaptation required novel theoretical developments to characterize the loss landscape of LoRA.

The main theorem then states that, under these assumptions, any second-order stationary point of the LoRA objective function will fall into one of two categories:

1. A global minimum with low rank and small magnitude, corresponding to $X_{\text{star}}$.
2. A spurious local minimum with high rank and large magnitude.

The proof then uses the implicit biases of LoRA training—zero initialization (which favors small-magnitude solutions) and weight decay (which favors low-rank solutions)—to argue that the optimization process is highly likely to converge to the first category. This is visualized as the training starting near the origin (zero update) and being regularized to stay within a "nice region" of the loss landscape where only the good, low-rank global minima exist, thus avoiding the "treacherous region" of high-rank spurious minima.

Experimental Setup & Results

▶ Watch: Prior work: LoRA has no spurious local minima (strong assumptions) (5:10)

The experimental component of this work primarily serves to empirically validate the key theoretical assumptions and the main theorem's predictions, rather than presenting novel state-of-the-art performance benchmarks. The focus is on understanding the fundamental behavior of the LoRA loss landscape.

The first set of experiments aimed to verify the newly introduced theoretical assumptions: restricted strong convexity (RSC) and restricted strong smoothness (RSS). While the talk doesn't detail the specific models, datasets, or hardware used for these validations, it states that these properties were empirically confirmed to hold in practice. This is a crucial finding, as it provides a practical basis for the theoretical framework built upon these assumptions. Without this empirical grounding, the theoretical results would remain abstract.

A particularly interesting observation from these validation experiments concerned the relationship between the LoRA rank parameter $R$ and the favorability of the loss landscape. The coefficients associated with RSC and RSS were found to become more favorable as the rank parameter $R$ becomes smaller. This implies that choosing a smaller LoRA rank (within the constraint that $R \ge R_{\text{star}}$) doesn't just reduce memory footprint and computational cost; it also contributes to a "nicer," more well-behaved loss landscape, potentially making optimization easier or more robust. This provides an additional, theoretical justification for preferring smaller LoRA ranks when possible.

The second and most compelling set of experiments directly validated the main theorem's prediction of two distinct scenarios for second-order stationary points. These experiments demonstrated the existence of the "loud failure" spurious local minima but also confirmed that standard LoRA training effectively avoids them.

1. Standard LoRA Initialization: When the LoRA factors were initialized according to common practice (random i.i.d. Gaussians for $A$ and all zeros for $B$), the training consistently converged to the global minimum. This outcome aligns with the "good" scenario, where LoRA successfully finds the optimal low-rank solution. This result provides strong empirical evidence supporting the theoretical claim that standard LoRA practices lead to desirable convergence.
2. Random Non-Zero Initialization: To explicitly demonstrate the existence of the "bad" spurious local minima, the experiments used a random non-zero initialization for the LoRA factors. Under this perturbed initialization, the training converged to a spurious local minimum. These spurious minima were characterized by having a large rank and large magnitude update, consistent with the "failing loudly" scenario described in the theorem. This experimental result is vital as it moves the concept of "bad" local minima from a theoretical possibility to an empirically observed phenomenon, while simultaneously highlighting why LoRA's default initialization is so effective at avoiding them.

The talk did not provide specific details on the datasets, model architectures (e.g., BERT, GPT-2), or hardware (e.g., GPU types, number of GPUs) used for these experiments. However, the qualitative results strongly support the theoretical claims, adding significant credibility to the refined understanding of LoRA's optimization landscape. The absence of specific performance metrics or comparisons against other fine-tuning methods underscores the work's foundational nature, prioritizing theoretical understanding and empirical validation of core mechanisms over demonstrating practical performance gains.

Practical Implications

▶ Watch: Limitations of prior work and new relaxed assumptions (6:05)

This theoretical work offers profound practical implications for anyone working with LoRA, from model builders to infrastructure teams and deployers.

Firstly, and perhaps most importantly, it provides strong theoretical confidence in LoRA's effectiveness. Practitioners have widely adopted LoRA due to its empirical success, but this work provides a rigorous explanation for why it works. Knowing that LoRA is provably guided towards a low-rank global minimum under reasonable assumptions validates its widespread use and solidifies its position as a robust parameter-efficient fine-tuning technique. This allows practitioners to deploy LoRA with greater assurance, understanding that its success is not merely a heuristic but grounded in solid optimization theory.

The work underscores the critical importance of LoRA's standard initialization strategy and the role of weight decay. The empirical results clearly demonstrated that while "bad" spurious local minima exist, the typical initialization (random $A$, zero $B$) coupled with weight decay effectively steers the optimization away from them. This means practitioners should adhere to these practices, as deviating from them (e.g., using random non-zero initialization for $B$) significantly increases the risk of converging to a suboptimal, high-rank solution with poor generalization. For infrastructure teams, this reinforces the need to ensure default LoRA implementations correctly apply these initialization schemes and weight decay.

The findings also provide a new perspective on LoRA rank selection. While the common understanding is that a smaller rank $R$ reduces memory and computational overhead, this research suggests an additional benefit: a more favorable (more convex-like) loss landscape for smaller $R$. This insight can inform hyperparameter tuning strategies. Practitioners might be encouraged to try the smallest possible $R$ that still allows for sufficient expressivity (i.e., $R \ge R_{\text{star}}$), as it could lead to faster convergence and more robust optimization, in addition to efficiency gains. This adds a theoretical dimension to the practical trade-offs involved in choosing $R$.

Furthermore, by moving beyond the NTK assumption, this work's analysis is more robust and applicable to a wider range of practical LoRA fine-tuning scenarios where the NTK assumption might not hold. This is particularly relevant for fine-tuning tasks that involve significant parameter changes or highly non-linear model behaviors, where prior NTK-based theories might have been less reliable. This broader applicability means the theoretical guarantees extend to more diverse real-world applications of LoRA.

Regarding trade-offs and limitations, the work acknowledges that the claim "it probably won't fail" is intuitive rather than formally proven. While the implicit biases strongly suggest avoiding "loud failures," a rigorous, probability-one proof for this aspect remains future work. Additionally, the core assumption of the existence of a low-rank global minimizer ($X_{\text{star}}$) is a prerequisite for the theory. The work does not delve into why fine-tuning tasks inherently admit such low-rank updates, which is an important open question. This means if a particular fine-tuning task truly requires a high-rank update for optimal performance, LoRA (by design) might be inherently limited, regardless of its convergence guarantees. However, for the vast majority of tasks where LoRA performs well, this work explains how it finds those low-rank solutions.

In essence, this research transforms LoRA from a highly effective empirical technique into one with strong theoretical backing, providing clearer guidelines for its successful deployment and offering deeper insights into its underlying mechanisms.

Key Takeaways

• LoRA training, under reasonable assumptions, is guaranteed to converge to a low-rank global minimum.
• If LoRA training does converge to a suboptimal solution, it will be a high-rank, large-magnitude spurious local minimum, a failure mode described as "failing loudly" and easily detectable.
• Standard LoRA practices, including zero initialization for the $B$ factor and the application of weight decay, act as crucial implicit biases that guide the optimization away from these "loud failure" spurious minima towards the desired low-rank global minimum.
• This work's analysis relies on weaker assumptions (restricted strong convexity and smoothness) than prior theoretical work that often depended on the restrictive Neural Tangent Kernel (NTK) assumption, making the findings more generally applicable.
• Empirical evidence confirms that the new assumptions hold in practice and that while spurious minima exist, standard LoRA initialization successfully avoids them.
• A surprising finding is that smaller LoRA ranks (R) not only offer memory and computational benefits but also lead to a more favorable (more convex-like) loss landscape, potentially facilitating easier optimization.

About the Speaker(s)

The talk was presented by Ernest Ryu, who is a co-author on this significant research. While his specific title or affiliation beyond "presenter" is not detailed in the transcript, his role in presenting and co-authoring such a rigorous theoretical work suggests a position as a senior researcher or faculty member in the field of machine learning and optimization.

Junsu Kim is highlighted as a key contributor to this work. At the time of the talk, he is an undergraduate student currently undertaking a research internship at Google DeepMind. He is actively applying for PhD programs, indicating a promising academic trajectory in AI/ML research.

Jaeyeon Kim is also a co-author of the presented work. The transcript does not provide specific biographical details about Jaeyeon Kim, but their involvement underscores the collaborative nature of this theoretical breakthrough.