r(d+k) for a 4096×12288 projection.
LoRA freezes the original weight matrix and learns two small factors whose product becomes the task-specific update. Change rank, scale, and data, then inspect capacity, memory, and approximation in 3D.
Open the matrix labA dense d×k update has d·k degrees of freedom. LoRA assumes useful adaptation lies in a lower-dimensional subspace, representing the update as B·A with rank r. The base model stays shareable while small adapters carry specialized behavior.
r(d+k) for a 4096×12288 projection.
Adapter parameters divided by d·k.
Target update energy captured by rank r.
Trainable factors only, before optimizer states.
The original weight W continues to transform the input. Gradients do not update W, preserving the reusable base checkpoint and reducing optimizer state.
h = W x + ΔW xA projects the input into r adapter coordinates; B projects those coordinates back to the output dimension. Rank limits the update's independent directions.
ΔW = (α / r) B AThe product can be added into W for zero adapter lookup overhead, or kept separate so one base model can serve many small task adapters. Multiple adapters are not automatically compositional; their updates may interfere.
W′ = W + (α / r) B AThe bottleneck cannot express all useful update directions. Training plateaus with structured residual energy.
Inspect fit before increasing epochs.Large α/r amplifies the adapter, potentially overwriting useful base behavior or destabilizing optimization.
Tune learning rate and scale together.A capable adapter can memorize examples while failing outside their wording and distribution.
Validate invariance, not training loss alone.