MMOOZZGG.com

Glossary

What is an autoencoder

In short

An autoencoder is a neural network that compresses data into a short code and then restores the data from that code. While doing this, it learns to pick out what matters in the data.

Machine learning · autoencoder

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you need to describe a drawing to a friend, but you may say only five words. You say: “red house, two windows, chimney”. From these words, your friend draws the house again, almost the same. To make this work, you must choose the main words and skip the small details. This program learns in a similar way: it squeezes a picture into a short note and draws it again from the note.

In plain words

The network has two halves. The first, the encoder, compresses the input, such as a photo or a row of sensor readings, into a short set of numbers. The second, the decoder, tries to restore the original from this set. It learns by itself, without labels: the error is the difference between the input and the restored copy. The narrow middle forces it to keep what matters and drop the noise. This is where the value comes from: if you show the model something unusual, it restores it badly, and a large error tells you the example is strange.

For business

Companies use this approach for three tasks. The first is anomaly detection. The model learns on the normal work of a machine, a network or payments, and everything it struggles to restore goes to a person for review. You hardly need labeled failures for this, which helps when the history has few breakdowns. The second is data compression and cleaning, such as removing noise from scans and recordings. The third is compact features for other models and for finding similar items. Costs are low: the model is small and runs on ordinary servers. The risk is in the alarm threshold: one that is too strict floods the team with false alerts, and one that is too loose misses a failure. Set the threshold together with the people who will handle the alerts.

For a specialist

An autoencoder is a neural network trained to reproduce its input: the encoder maps x to a latent representation z, the decoder restores an approximation of x from z, and the loss function measures the reconstruction error, for example the mean squared error. To stop the network from learning the identity function, a constraint is added: a narrow bottleneck (undercomplete), sparse activations (sparse), noise on the input or masking. The denoising variant learns to restore a clean input from a corrupted one; Vincent and coauthors (JMLR, 2010) stacked such blocks into deep networks and got features useful for later classification. The masked variant MAE (He and coauthors, 2021) hides a large share of image patches, such as 75%, and restores the pixels; it is a self-supervised learning method for vision. The variational version adds a probabilistic model of the latent space and can generate data. Sparse autoencoders are used in interpretability: Bricken and coauthors (Anthropic, 2023) broke down the activations of a 512-neuron MLP layer of a one-layer transformer into thousands of understandable features.

Example

An internet provider trained an autoencoder on several months of normal network traffic. When a packet flow suddenly restores with a large error, the engineer on duty gets an alert and checks for an equipment failure or an attack.

Related terms

Questions and answers

Why do you need an autoencoder if it just copies the input? The copy itself is not needed. The value is in the narrow middle: the network must compress the data and keep only what matters.

The compressed code and the rebuild error give features for anomaly detection, compression and other models.

How is a plain autoencoder different from a variational one? A plain one learns one code for each example and is good for compression, cleaning and anomaly detection. A variational one describes codes as a distribution, so you can draw new examples from its space.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us