MMOOZZGG.com

Glossary

What is an encoder in a neural network

In short

An encoder is the part of a neural network that reads the whole input and turns it into a set of numbers, which is handy for search, comparison or later generation.

Language models · encoder

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine you read a long book and tell a friend its main idea in three sentences. Your friend never opened the book, but from your summary understands what it is about and decides whether it is interesting. A neural network has a part that makes such a summary, only not in words but as a secret list of numbers that other parts of the network understand.

In plain words

An encoder takes text, an image or sound and reads it fully, linking each word with all the others, both to the left and to the right. The output is a set of numbers that holds the meaning in compressed form. Other programs then work with these numbers: they compare two customer questions and see that both are about the same thing, put an email into the right category, or find a similar document. The encoder itself does not write new text. For example, a knowledge base search finds the guide “How to request a refund” for the query “I want my money back”, because the encoder gave them close numbers.

For business

Encoder models are almost invisible, yet they support document search, sorting of customer requests and checks for duplicates. They are much smaller than chat models, run on an ordinary server and answer in a fraction of a second. So it pays to run them inside the company: texts stay within your network, and the cost of a request is close to zero. Their weak point is language: a model trained mostly on English texts tells apart the meaning of Russian wording and industry terms less well. Before you launch search, collect a hundred real staff queries with correct answers and count how often the right document lands in the top three. Run this check before you choose a model for an AI assistant over documents.

For a specialist

An encoder maps an input sequence to a sequence of hidden states. In the encoder-decoder scheme of Cho and coauthors (2014), one recurrent network compressed a phrase into a fixed-length vector, and a second one unfolded it into the output sequence. In the transformer of Vaswani and coauthors (2017), the encoder consists of self-attention layers without a mask: each token sees the whole context in both directions. The class of encoder-only models is represented by BERT (Devlin and coauthors, 2018). It is pretrained with masked language modeling, which hides 15% of tokens and teaches the model to restore them; BERT-base has 110 million parameters. Such models are fine-tuned for classification, entity extraction and ranking. Contrastive training turns them into embedding models, and cross-encoders score a query and document pair for reranking. They cannot generate coherent text, so for an answer they are joined with a decoder.

Do not confuse. Do not confuse it with an encoder in machine tools and motors, where the word means a sensor of shaft angle.

Example

A support team gets a thousand emails a day. A small encoder model sorts each one in a fraction of a second into the topic “delivery”, “refund” or “payment”, and the email goes straight to the right employee.

Related terms

Questions and answers

How is an encoder different from a decoder? An encoder reads the whole input at once and turns it into a numeric description of meaning. A decoder writes the output one token at a time and sees only what is already written.

The first is for search and classification, the second for text generation.

Where are encoder models used? In knowledge base search, in sorting emails and requests, in finding duplicates and in reranking search results. They are fast and cheap, so they often run on a company’s own servers.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us