Glossary
QLoRA: fine-tuning on one graphics card
In short
QLoRA is a way to fine-tune a large language model on a single graphics card (GPU). The base model is compressed to 4 bits, and small LoRA adapters are trained on top of it.
Language models · QLoRA
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a huge puzzle that does not fit on your desk. You pack its pieces into tight little boxes, and it takes up little space. Then you glue a few of your own pictures on top. The puzzle is the same, but now it has your drawings, and it all fits on a small desk.
In plain words
To teach a large AI model something new, you usually need many powerful graphics cards with a lot of memory. This method saves memory with two tricks. First, the model’s numbers are written more roughly, several times more compactly, and frozen. Then small trainable add-ons are placed next to them, and only these change. The model loses almost no quality, and training fits on one graphics card. It is like a thick reference book rewritten in small print, with your own bookmarks glued in.
For business
The method lowers the entry cost of your own model. Fine-tuning that once needed a rented cluster now fits on one server with a professional graphics card. This matters when you cannot send data to an outside service and the model must run inside your network. The cost depends on GPU memory, model size and the number of training runs, and most of the work goes into preparing examples. The limits: training can be slower than with an uncompressed model, and you must test the final model on your tasks, because compression sometimes breaks rare skills. Which graphics card can handle the model size you need is calculated in advance, by memory and price.
For a specialist
QLoRA (Dettmers, Pagnoni, Holtzman and Zettlemoyer, 2023) combines quantization with low-rank adapters. Gradients pass through the frozen model, whose weights are stored in 4 bits, into the trainable LoRA matrices. At each use the weights are unpacked to BFloat16, and the multiplication runs in 16 bits. The authors report that the quality of 16-bit fine-tuning is preserved. The work has three innovations. The 4-bit NormalFloat (NF4) data type is argued to be theoretically optimal for normally distributed weights. Double quantization also compresses the quantization constants. Paged optimizers smooth out peaks in memory use. The authors fine-tuned more than 1,000 models. Their Guanaco family reached 99.3% of the level of ChatGPT on the Vicuna benchmark after 24 hours of training on one GPU. They published the models, the code and CUDA kernels for 4-bit training. In the PEFT library the method is assembled from 4-bit loading through bitsandbytes and LoRA adapters. The weak points are the overhead of dequantization and the need to check quality after merging the adapter.
Example
A logistics company fine-tuned an open model on its own templates of letters to carriers. A server with one graphics card finished the job over a weekend, and the model stayed inside the corporate network.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map