LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

Yuanhe Zhang, Fanghui Liu, Yudong Chen

International Conference on Machine Learning 2025 · Oral

This talk, presented by Fanghui Liu at ICML 2025, introduces **LoRA-One**, a novel approach to fine-tuning **Large Language Models (LLMs)** that redefines the trade-off between performance and efficiency. The core premise of LoRA-One is that a single computation of the full gradient, combined with a theory-grounded initialization strategy, can be sufficient for effectively fine-tuning LLMs using **Low-Rank Adaptation (LoRA)**. The work is a collaboration with Yuanhe Zhang (lead PhD student) and Yudong Chen.

AI review

LoRA-One offers a genuine theoretical contribution — a subspace alignment theorem explaining where trained LoRA matrices end up — and parlays that into a principled spectral initialization. The linear model proof is credible and the empirical correction of LoRA-GA is valuable. What keeps this at three stars is that the theoretical machinery is established for linear models and then hand-waved to nonlinear ones, the experimental footprint is modest (CoLA and MRPC are small benchmarks, and the Llama 2 results lack quantitative detail in the write-up), and the core algorithmic idea — initialize…