MMOOZZGG.com

Glossary

Recurrent neural network (RNN)

In short

A recurrent neural network (RNN) reads data one element at a time and keeps an internal memory of what it has read. This makes it a fit for text, speech and series of numbers.

Machine learning · recurrent neural network

MMOOZZGG.com editorsChecked: 1 min read

For a child

When someone reads you a fairy tale, you don’t forget the beginning while you listen to the end. You remember that the gingerbread man ran away from the old woman and from the hare, so you understand why the fox plays tricks. This kind of program also listens to a story word by word and keeps the main points of the past in mind. But it remembers very long stories poorly.

In plain words

A regular neural network looks at all the data at once. A recurrent one reads it in order: word after word, second after second of a recording, day after day of sales. After each step it updates its memory, a short summary of what it has read, and passes it on. This is how it understands that a “not” at the start of a phrase changes the meaning of its end. Its weak spot is long chains: early details slowly fade. The next-word suggestion on a phone keyboard works in a similar way: it takes into account the words you have already typed.

For business

For new text projects, teams rarely choose these networks now: transformers and large language models have taken their place. But they stay in working systems and where speed and modest resources matter: demand and load forecasts from time series, equipment sensors, and speech recognition on the device. The model is small, runs on a regular processor and needs no expensive GPUs. If a contractor offers this architecture, ask what they compared it with: gradient boosting on features and a transformer. The risk is a long history: the network gives little weight to events several months old. Choose the right forecasting method on the company’s historical data before the rollout.

For a specialist

A recurrent neural network (RNN) processes a sequence step by step: the hidden state h_t is computed from the input x_t and the earlier state h_{t-1}, with the same weights at every step. It is trained by backpropagation through time, and on long sequences the gradients fade away or blow up. LSTM (long short-term memory; Hochreiter and Schmidhuber, Neural Computation, 1997) solves this with a memory cell that has a constant error flow and multiplicative gates; the authors showed links across intervals of more than 1,000 steps. GRU (Cho and co-authors, 2014) simplifies the cell to two gates, reset and update. Variants include bidirectional and multilayer RNNs and the encoder-decoder scheme for translation. Sequential computation is hard to run in parallel, and the transformer (Vaswani and co-authors, 2017) dropped recurrence entirely in favor of attention. State space models and linear recurrent layers bring RNN ideas back for long contexts.

Example

A power supply company forecasts consumption one day ahead. An LSTM reads the hourly history of load and weather for past weeks and produces a forecast that the dispatcher uses to plan capacity purchases.

Related terms

Questions and answers

How is a recurrent neural network different from a transformer? A recurrent network reads a sequence step by step and keeps its memory in one state vector. A transformer looks at all elements at once through attention, so it trains faster on GPUs and holds distant links more reliably.
What is LSTM? LSTM is a type of recurrent network with a memory cell and gates that decide what to remember, what to forget and what to pass out. This cell helps the network take into account events that are far apart in time.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us