Glossary
What is a decoder in a transformer?
In short
A decoder in a transformer is the part of a neural network that creates an output sequence one element at a time, each time relying on what is already written.
Language models · decoder
MMOOZZGG.com editorsChecked: 2 min read
For a child
Remember the game where each player adds one word to a story in turn? You hear the start and think of the next word so the story does not fall apart. Then the next player does the same. The part of a neural network that writes answers plays this game with itself: word by word, until the story ends.
In plain words
When a chatbot types an answer, a decoder is at work. It looks at your question and at the part of the answer already written. It estimates what the next piece of text could be, picks one and adds it to the end. Then it repeats everything again, now with the new piece. It is not allowed to look ahead: in training, future words are hidden, or the model would learn to copy. That is why a long answer takes longer than a short one, and the text appears on the screen bit by bit, not all at once.
For business
Almost all models that write texts, code and emails are built only on a decoder. This explains the properties a business already knows. You pay for every generated token, so a wordy model costs money. Typing speed is limited because tokens come one after another, and long reports need time and powerful graphics cards. The model picks a plausible continuation, not a checked fact, which is where made-up answers come from. If employees need a short answer, limit the length and ask for a table format: this cuts both the bill and the waiting time. Team training covers how to set such tasks for generative AI.
For a specialist
The decoder in the transformer of Vaswani and co-authors (2017) consists of layers with masked self-attention. The mask stops each position from looking at the later ones and keeps the auto-regressive property. In the original encoder-decoder scheme, each layer also has cross-attention to the encoder output. T5 (Raffel and co-authors, 2019) works this way and turns any task into a “text in, text out” format. Decoder-only models drop cross-attention and learn next-token prediction on raw text; the GPT family and most modern LLMs are built this way. Generation splits into the prefill of the prompt and step-by-step decoding, and the KV cache saves recomputing the keys and values of past tokens. Decoding strategies choose the next token from the distribution: greedy, beam search, and sampling with top-k and top-p.
Do not confuse. Not to be confused with a TV decoder, a set-top box that receives a digital signal.
Example
A manager asks the model to write a letter to a client. The first word, “Good”, appears almost at once, and the rest follow one after another: the decoder outputs one token at a time and rereads everything written each time.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map