Glossary
LoRA fine tuning: cheap adaptation of large models
In short
LoRA (Low-Rank Adaptation) is a cheap way to fine-tune a large neural network: the main weights stay unchanged, and a small add-on is trained that adapts the model to a task or a style.
Images, video, audio · low-rank adaptation
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a thick encyclopedia in which you cannot erase anything. To add your own knowledge, you stick small colored notes on the pages. The book stays the same, but with the notes it tells things your way. The notes are easy to remove and replace. A neural network can be extended in the same way, without rewriting all of it.
In plain words
A large model has billions of numbers, and retraining all of them is slow and costly. The trick is to freeze the original numbers and add next to them a small new set, many times smaller. Only that set is trained. The result is a separate light file that plugs into the model like an attachment. One model can have many such attachments: one draws in the company’s brand style, another answers in the tone of your support team, a third knows industry terms.
For business
LoRA fine tuning makes custom tuning of a model affordable for a company without its own data center: training runs on one or a few GPUs, and the result is small and stored apart from the base model. This gives you a model that writes in your brand style, understands internal terms or draws your product and character. The cost depends on the model size, the number of examples and how often you will need to retrain after changes. The risks: the adapter inherits the license and the weaknesses of the base model, bad examples lock in mistakes, and third-party adapters from the internet may infringe other people’s rights. First check whether RAG or a good instruction solves the task, and only then pay for training. Companies usually keep such adapters on their own servers.
For a specialist
Low-Rank Adaptation was proposed by Hu et al. (2021): the weights W of a pretrained model are frozen, and the update is written as the product of two low-rank matrices, ΔW = BA, where the rank r is much smaller than the layer dimension. Only A and B are trained, usually in the transformer’s attention projections. According to the authors, for GPT-3 175B the number of trainable parameters fell about 10,000 times and GPU memory needs fell threefold compared with full fine-tuning, at comparable quality. After training, the product BA is merged into the original weights, so there is no inference latency, unlike adapter layers. QLoRA (Dettmers et al., 2023) trains such adapters on top of a model quantized to 4 bits (the NF4 format), which made it possible to fine-tune a 65-billion-parameter model on one GPU with 48 GB of memory. In diffusion models, low-rank adapters go into the attention layers of the denoiser, including cross-attention, for styles and characters. The method belongs to the PEFT family.
Example
An interior design studio trained an adapter on a hundred of its projects. Now the image generator draws sketches in the studio’s recognizable style, and designers refine the strongest drafts.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map