Paper
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
arXiv:2106.09685 · 0▲ · cs.CL, cs.AI, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
LoRA (Low-Rank Adaptation) freezes a pre-trained model's weights and injects trainable rank-decomposition matrices (a pair of small matrices B and A whose product approximates the weight update) into each Transformer layer, so only those small matrices are trained. In their Transformer experiments the authors apply it mostly to the query and value attention projection matrices.
Why it matters
You can adapt very large models to new tasks while training a tiny fraction of the parameters, which cuts GPU memory and storage sharply and lets you swap task-specific weights in and out cheaply. Because the low-rank matrices can be merged back into the frozen weights at deployment, there is no extra inference latency, unlike adapter layers.
Practical takeaway
You can fine-tune a large model by storing per-task LoRA weights of a few tens of MB instead of full multi-hundred-GB checkpoints, keeping one frozen base model in memory and switching tasks by swapping the A and B matrices. Watch for rank r as the main knob: values as low as 1 to 8 worked for their tasks, though the optimal rank varies and higher ranks may be needed for tasks far from the pre-training distribution.
Key result
On GPT-3 175B, LoRA reduced trainable parameters by up to 10,000x and training VRAM by 3x (1.2TB to 350GB), while matching or exceeding full fine-tuning: 73.4% vs 73.8% on WikiSQL and 91.7% vs 89.5% on MNLI-matched accuracy. Caveat: GPT-3 results report standard deviation only per task rather than per entry due to training cost, and MLP layers were left unadapted.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.