LoRA

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: instead of updating all of a model's weights, you freeze the pretrained weights and train a small pair of low-rank matrices that get added to selected layers.

The idea rests on the observation that the weight update during fine-tuning has low intrinsic rank. So for a weight matrix W of shape (d, d), you approximate the change as W + BA, where A is (r, d) and B is (d, r), with r much smaller than d. Only A and B are trained; W stays fixed.

Concretely, for a 4096 x 4096 attention projection with rank r = 16, you go from about 16.8M trainable parameters to 2 * 4096 * 16 = 131K, roughly a 128x reduction. You typically apply LoRA to the query and value projections, scale the update by alpha / r, and can merge BA back into W at inference so there is no added latency.

Practical consequences:

• Checkpoints are tiny (megabytes), so you can keep many task-specific adapters and swap them at load time.
• Lower memory during training, since you skip optimizer state for the frozen base.

Think of it like patching a large binary with a small diff rather than recompiling the whole thing.

LoRA is relevant to a setup like Orca's, where a frozen backbone (a Qwen VLM) is adapted for downstream tasks. The paper trains separate small decoders per task, but LoRA is the standard alternative when you want to nudge the backbone's internal representations cheaply rather than only reading from them. QLoRA extends this by keeping the frozen base in 4-bit precision.