Glossary
What is an encoder in a neural network
In short
An encoder is the part of a neural network that reads the whole input and turns it into a set of numbers, which is handy for search, comparison or later generation.
Language models · encoder
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you read a long book and tell a friend its main idea in three sentences. Your friend never opened the book, but from your summary understands what it is about and decides whether it is interesting. A neural network has a part that makes such a summary, only not in words but as a secret list of numbers that other parts of the network understand.
In plain words
An encoder takes text, an image or sound and reads it fully, linking each word with all the others, both to the left and to the right. The output is a set of numbers that holds the meaning in compressed form. Other programs then work with these numbers: they compare two customer questions and see that both are about the same thing, put an email into the right category, or find a similar document. The encoder itself does not write new text. For example, a knowledge base search finds the guide “How to request a refund” for the query “I want my money back”, because the encoder gave them close numbers.
For business
Encoder models are almost invisible, yet they support document search, sorting of customer requests and checks for duplicates. They are much smaller than chat models, run on an ordinary server and answer in a fraction of a second. So it pays to run them inside the company: texts stay within your network, and the cost of a request is close to zero. Their weak point is language: a model trained mostly on English texts tells apart the meaning of Russian wording and industry terms less well. Before you launch search, collect a hundred real staff queries with correct answers and count how often the right document lands in the top three. Run this check before you choose a model for an AI assistant over documents.
For a specialist
An encoder maps an input sequence to a sequence of hidden states. In the encoder-decoder scheme of Cho and coauthors (2014), one recurrent network compressed a phrase into a fixed-length vector, and a second one unfolded it into the output sequence. In the transformer of Vaswani and coauthors (2017), the encoder consists of self-attention layers without a mask: each token sees the whole context in both directions. The class of encoder-only models is represented by BERT (Devlin and coauthors, 2018). It is pretrained with masked language modeling, which hides 15% of tokens and teaches the model to restore them; BERT-base has 110 million parameters. Such models are fine-tuned for classification, entity extraction and ranking. Contrastive training turns them into embedding models, and cross-encoders score a query and document pair for reranking. They cannot generate coherent text, so for an answer they are joined with a decoder.
Do not confuse. Do not confuse it with an encoder in machine tools and motors, where the word means a sensor of shaft angle.
Example
A support team gets a thousand emails a day. A small encoder model sorts each one in a fraction of a second into the topic “delivery”, “refund” or “payment”, and the email goes straight to the right employee.
Related terms
Questions and answers
How is an encoder different from a decoder?
Sources
Where companies use it
Not sure whether your company needs this?
AI solutions map