MMOOZZGG.com

Glossary

What is the GGUF format?

In short

The GGUF format is a file format for language models that stores the weights and all the data needed to run the model in one file, most often in compressed form for local use.

Infrastructure · GGUF

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine you are building a construction set. Sometimes the pieces are in one box, the instructions in another, and the stickers are lost. And sometimes everything is in one box: pieces, plan and stickers. Open it and build. This file is like a box where everything for a smart helper is packed together, so it is easy to start even on a home computer.

In plain words

When you download a neural network to run it yourself, you get a file with weights: billions of numbers that make up the model. GGUF files are handy because they hold both the numbers and the model description: its structure, the token dictionary and the context length. The program does not need to look for extra files. Such models are usually compressed already, so they fit in a laptop’s memory. For example, you pick a file marked Q4_K_M on a model site, open it in LM Studio or Ollama and soon talk to the model without the internet.

For business

For a company, the GGUF format means a fast path to a pilot on its own hardware: you can download an open model ready to run, with no conversion and no machine learning engineer at the start. Choosing a file means choosing between quality and hardware cost: a more compressed version fits a cheaper graphics card or a Mac but handles complex tasks worse. Risks: not only model authors publish such files, but also third parties, so take them from trusted repositories and check the license of the original model. For server loads with dozens of users at once, teams often choose other engines and formats. Compare the options on your own tasks before you buy hardware.

For a specialist

GGUF is a binary file format for models for inference on GGML and compatible engines; the author of the llama.cpp engine developed it. The format replaced GGML, GGMF and GGJT. The main difference from GGJT: metadata, formerly hyperparameters, is stored as typed key-value pairs, so new fields are added without breaking old files. The goals of the specification: single-file deployment, mmap compatibility for fast loading, and the full set of data needed to run inside the file. Structure: a header with a magic number, a version and counters, metadata (architecture, context length, tokenizer), tensor descriptions with type and offset, alignment (32 bytes by default) and the data. Quantization types: K-quants from Q2_K to Q6_K, IQ variants with an importance matrix, and the legacy Q4_0 and Q8_0; for example, Q4_K gives 4.5 bits per weight. Unlike safetensors, which holds only tensors, GGUF has standardized metadata.

Example

A purchasing team wants to check whether an open model can handle reviewing supplier offers. An analyst downloads two files of the same model with different compression, runs twenty real emails through each and sees that the lighter version mixes up amounts. The company picks the larger version and a graphics card with more memory.

Related terms

Questions and answers

How is GGUF different from safetensors? Safetensors holds only the weight tensors, and the model description sits in separate files. A GGUF file holds both the weights and the metadata: architecture, tokenizer and context length, so the model runs from a single file.
What do letters like Q4_K_M in a file name mean? They name the quantization type: Q4 means about 4 bits per weight, K points to the K-quant family, and the last letter separates variants inside the family. The smaller the number after Q, the smaller the file and the more the answer quality may drop.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us