From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control

Chi Zhang, REN Lianhai, Jingpu Cheng, Qianxiao Li

International Conference on Machine Learning 2025 · Oral

This talk, presented by Chi Zhang and co-authors REN Lianhai, Jingpu Cheng, and Qianxiao Li from the National University of Singapore, introduces a novel perspective on **Parameter-Efficient Fine-Tuning (PEFT)** algorithms, particularly **Low-Rank Adaptation (LoRA)**. Moving beyond the conventional understanding of LoRA as a weight-tuning technique, the authors propose a reinterpretation through the lens of **classical control theory**. This shift in perspective leads to the development of a **state-based fine-tuning** approach utilizing **parallel control**, which significantly reduces memory consumption and training time for large language models.

AI review

A competent systems-engineering contribution that wraps a practical memory optimization — skip the intermediate activation storage by rerouting gradients through a parallel shortcut — in the language of classical control theory. The core optimization insight is real and the empirical results are plausible, but the theoretical framework is more analogical than formal, and the memory saving mechanism is less novel than the framing suggests. Worth attending for practitioners working on LoRA-variant design; less essential for those seeking new theoretical foundations.