AI glossary
LLM terms and language model glossary
LLM terms describe how large language models are built, trained and run. This category covers tokens, context, training stages, tuning methods, generation settings and the main quality measures.
MMOOZZGG.com editors·Checked: ·10 min read
Terms in this category
- tokenA token is a piece of text that a language model works with: a word, part of a word, or a symbol.
- large language modelA large language model (LLM) is a neural network trained on a huge amount of text. It writes, summarizes and analyzes text on request.
- embeddingAn embedding is a list of numbers by which a computer understands the meaning of a text or a picture and finds similar ones.
- LLM routerAn LLM router (model router) is a program that decides, for each request, which language model should answer it: simple questions go to a cheap and fast model, hard ones to a more powerful model.
- AI hallucinationAn AI hallucination is an answer from a neural network that sounds confident but contains invented or wrong facts.
- foundation modelA foundation model is a large neural network trained on a huge amount of general data, which teams then adapt to many specific tasks.
- AI benchmarkTo benchmark AI models means to compare them on one standard set of tasks with known answers. That set is called an AI benchmark.
- model weightsModel weights are the numbers inside a neural network that set the strength of the links between its elements and store everything the model learned in training.
- vision language modelA vision language model (VLM) is a neural network that takes an image and text and answers in words: it describes a photo, reads a document, or answers questions about a picture.
- AI alignmentAI alignment is the tuning of an AI system so that it behaves as people intend: helpful, honest and harmless.
- test-time computeTest-time compute is the extra calculation a model spends while it answers, for example on reasoning or on checking several options, to solve a task more accurately.
- knowledge cutoffA knowledge cutoff is the point in time up to which the texts for training a language model were collected. The model does not know about later events by itself.
- decoderA decoder in a transformer is the part of a neural network that creates an output sequence one element at a time, each time relying on what is already written.
- knowledge distillationKnowledge distillation is a way to train a small model to repeat the answers of a large one, to get almost the same result cheaper and faster.
- long contextLong context is the ability of a language model to take in and use a very large amount of text in one request: thick documents, a long correspondence or a whole code repository.
- model fine-tuningModel fine-tuning is the adjustment of an already trained model on your own examples, so that it performs a specific task more accurately or writes in the required style.
- scaling lawsScaling laws in AI are observed relations by which the quality of a language model rises in a predictable way as the number of parameters, the amount of data and the compute grow.
- instruction tuningInstruction tuning is the further training of a language model on pairs of “instruction and correct answer”. After it, the model carries out requests instead of just continuing the text.
- AI inferenceAI inference is the work of an already trained model when it receives a request and computes an answer, unlike training, where the model was only being tuned.
- LLM quantizationLLM quantization is a way to compress a neural network by storing its numbers with lower precision, so the model takes less memory and runs faster.
- constitutional AIConstitutional AI is a method from Anthropic in which a model is trained for safe behavior from a written list of principles: it criticizes and corrects its own answers, and people only write the rules.
- context windowThe context window is the amount of text, counted in tokens, that a language model sees at one time: instructions, documents, the question and its own answer.
- prompt cachingPrompt caching is a technique in which the model provider saves the already processed start of a request and takes it from memory when it repeats: the answer arrives faster and costs less.
- LLM latencyLLM latency is the delay between a request to a model and its answer: how long you wait for the first word and how long the whole answer takes.
- logitsLogits are raw numeric scores that a model gives to each possible answer, for example to each token of its vocabulary, before they are turned into probabilities.
- small language modelA small language model (SLM) is a compact language model with a few billion parameters or fewer that runs on an ordinary server, a laptop or a phone.
- attention mechanismThe attention mechanism is the way a neural network decides, for each word, which other words of the text to rely on and how strongly.
- world modelA world model is an internal model of the environment inside a neural network. It predicts how the world will change after an action and lets an agent plan before it acts.
- open weights modelAn open weights model is a neural network whose parameter files you can download and run on your own hardware.
- open source AIOpen source AI is an approach in which a model is published so that anyone can freely use, study, change and share it: with the weights, the code and information about the training data.
- model evaluationModel evaluation (evals) is a systematic check of AI answers on a set of your company’s tasks, to learn how accurate and safe the AI is and whether it is ready for work.
- model parametersModel parameters are the numbers inside a neural network that it sets during training and then uses to compute every answer.
- perplexityPerplexity is a quality metric for a language model that shows how “surprised” the model is by a text: the lower the value, the better the model predicts the next tokens.
- positional encodingPositional encoding is a way to tell a transformer the order of words: it adds information about the place of each token in the text to the description of that token.
- pretrainingPretraining is the first large stage in building a language model, in which it learns to predict the continuation of a text from a huge collection of books, websites and code.
- reasoning modelA reasoning model is a language model that writes out its own steps of the solution before it answers, and spends more time and computing on a hard task.
- mixture of expertsA mixture of experts (MoE) is a neural network design in which only a small part of the model works on each word. A component called the router picks that part.
- LLM temperatureLLM temperature is a generation setting that controls how random a language model’s answer is: with a low value the answers are more predictable, with a high value more varied.
- tokenizationTokenization is the splitting of text into tokens: pieces of words and symbols that a language model turns into numbers and only then reads.
- transformerA transformer model is a neural network architecture that reads all the text at once. It uses a mechanism called attention to decide which words matter for understanding each word.
- frontier modelA frontier model is a general-purpose AI model at the leading edge of ability, equal to the most powerful systems of its time or better.
- chain of thoughtChain of thought is a step-by-step solution that a language model writes before its answer, so that it handles a hard task more accurately.
- chat modelA chat model is a language model that has been tuned to hold a dialog. It understands the roles of the speakers, follows instructions, and answers each message, instead of only continuing a text.
- emergent abilitiesEmergent abilities in LLMs are skills that small language models lack and large ones show, and that could not be predicted from the results of smaller models.
- encoderAn encoder is the part of a neural network that reads the whole input and turns it into a set of numbers, which is handy for search, comparison or later generation.
- BPEByte pair encoding (BPE) is a tokenization algorithm. It builds a model’s vocabulary by merging the most frequent pairs of symbols into new units, again and again.
- DPODirect preference optimization (DPO) is a method of fine-tuning a language model on pairs of answers, a good one and a bad one, that teaches it to answer the way people prefer, without a separate reward model.
- KV cacheThe KV cache is the memory in which a language model keeps intermediate results (attention keys and values) for tokens it has already processed, so it does not recompute them for every new token.
- LLM as a judgeLLM as a judge (LLM-as-a-judge) is a way to evaluate the answers of neural networks with another language model, which gives scores by set criteria or picks the better of two answers.
- PEFTParameter-efficient fine-tuning (PEFT) is a family of methods that adapt a large model to a task by training only a small share of its parameters.
- QLoRAQLoRA is a way to fine-tune a large language model on a single graphics card (GPU). The base model is compressed to 4 bits, and small LoRA adapters are trained on top of it.
- RLHFRLHF (reinforcement learning from human feedback) is a stage of training a language model in which people compare its answers and the model learns to give the answers they prefer.
- SFTSupervised fine-tuning (SFT) is a stage at which a language model is trained further on pairs of a request and a model answer, so that it follows instructions in the needed format.
- softmaxThe softmax function turns a set of any numbers into probabilities: every value becomes a number between zero and one, and all values add up to one.
- top-kTop-k sampling is a generation setting in which a language model picks the next token only from the k most likely options at each step.
- top-pTop-p sampling, also called nucleus sampling, is a generation setting in which a model picks the next token only among the most likely options whose total probability reaches a threshold p.