MMOOZZGG.com

Glossary

LoRA fine tuning: cheap adaptation of large models

In short

LoRA (Low-Rank Adaptation) is a cheap way to fine-tune a large neural network: the main weights stay unchanged, and a small add-on is trained that adapts the model to a task or a style.

Images, video, audio · low-rank adaptation

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine a thick encyclopedia in which you cannot erase anything. To add your own knowledge, you stick small colored notes on the pages. The book stays the same, but with the notes it tells things your way. The notes are easy to remove and replace. A neural network can be extended in the same way, without rewriting all of it.

In plain words

A large model has billions of numbers, and retraining all of them is slow and costly. The trick is to freeze the original numbers and add next to them a small new set, many times smaller. Only that set is trained. The result is a separate light file that plugs into the model like an attachment. One model can have many such attachments: one draws in the company’s brand style, another answers in the tone of your support team, a third knows industry terms.

For business

LoRA fine tuning makes custom tuning of a model affordable for a company without its own data center: training runs on one or a few GPUs, and the result is small and stored apart from the base model. This gives you a model that writes in your brand style, understands internal terms or draws your product and character. The cost depends on the model size, the number of examples and how often you will need to retrain after changes. The risks: the adapter inherits the license and the weaknesses of the base model, bad examples lock in mistakes, and third-party adapters from the internet may infringe other people’s rights. First check whether RAG or a good instruction solves the task, and only then pay for training. Companies usually keep such adapters on their own servers.

For a specialist

Low-Rank Adaptation was proposed by Hu et al. (2021): the weights W of a pretrained model are frozen, and the update is written as the product of two low-rank matrices, ΔW = BA, where the rank r is much smaller than the layer dimension. Only A and B are trained, usually in the transformer’s attention projections. According to the authors, for GPT-3 175B the number of trainable parameters fell about 10,000 times and GPU memory needs fell threefold compared with full fine-tuning, at comparable quality. After training, the product BA is merged into the original weights, so there is no inference latency, unlike adapter layers. QLoRA (Dettmers et al., 2023) trains such adapters on top of a model quantized to 4 bits (the NF4 format), which made it possible to fine-tune a 65-billion-parameter model on one GPU with 48 GB of memory. In diffusion models, low-rank adapters go into the attention layers of the denoiser, including cross-attention, for styles and characters. The method belongs to the PEFT family.

Example

An interior design studio trained an adapter on a hundred of its projects. Now the image generator draws sketches in the studio’s recognizable style, and designers refine the strongest drafts.

Related terms

Questions and answers

What is LoRA in a neural network? It is a small extra file for a model that changes its behavior: the style of images, the tone of text or knowledge of a topic. The model itself is not retrained; only this add-on learns.
How is LoRA different from full fine-tuning? Full fine-tuning changes all the weights of the model, which needs a lot of memory and produces a heavy copy. An adapter trains a small add-on and keeps the original model, so it costs less and lets you keep many tuned versions.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us