LoRA / QLoRA (parameter-efficient fine-tuning)

Editorial record

Plain-language summary

Instead of updating a weight matrix directly, the method learns a low-rank product added to it, so only a tiny fraction of parameters are trained and the small adapters can be stored per task and swapped in. Because the update can be merged back into the base weights after training, there is no extra latency at inference, unlike adapter layers. On GPT-3 175B it reduced trainable parameters by roughly ten-thousand-fold while matching full fine-tuning quality, making per-task customization of large models cheap and modular.