Glossary
What is an activation function in a neural network?
In short
An activation function is a simple nonlinear rule inside an artificial neuron that decides which signal, and how strong, the neuron passes on.
Math of AI · activation function
MMOOZZGG.com editorsChecked: 1 min read
For a child
Think of a water tap at home. If you turn the handle just a little, no water comes out. Turn it more, and the stream gets bigger and bigger. Every tiny counter inside a smart program has a tap like this too: it decides whether to let a signal through and how strong it should be. Without these taps, the program could do very little.
In plain words
An artificial neuron adds up the incoming signals, multiplying each by its own weight. If it stopped there, the whole network, however many layers it has, would reduce to one simple formula and could not describe complex patterns. So after the sum comes a small switch. A common version is very simple: it turns a negative number into zero and lets a positive one pass as it is. Thanks to such bends, the network can draw a border of any complex shape between a cat and a dog in a photo. It is like wire: you can bend any figure from short straight pieces with bends, but not from one straight line.
For business
A manager does not have to choose this rule: the model developers build it in. But it indirectly affects speed and price. If a contractor proposes to train your own network from scratch, the architecture, including this element, affects the time and the budget of experiments. If you take a ready model, the choice is already made, and you pay for answers, not for experiments. A risk appears when you move the model to other hardware or another format: a rare variant of the function may not be supported in accelerated mode, and answers become slower. Test this on the target server before launch. How AI works in a company, level by level, is covered on a separate page.
For a specialist
An activation function is a nonlinear mapping applied to the output of a neuron’s linear transform: y = f(Wx + b). The backpropagation paper by Rumelhart, Hinton and Williams (Nature, 1986) used the logistic sigmoid 1/(1+e^(-x)); it and the hyperbolic tangent (tanh) have a small derivative at the edges, which makes deep networks hard to train. Nair and Hinton (ICML 2010) used rectified linear units (ReLU) in restricted Boltzmann machines, and Glorot, Bordes and Bengio (AISTATS 2011) showed that deep ReLU networks train as well as tanh networks without unsupervised pretraining and give sparse activations. GELU (Hendrycks and Gimpel, 2016) is defined as x·Φ(x), where Φ is the cumulative distribution function of the standard normal distribution, and it weights the input by its value instead of cutting it by sign. Shazeer (2020) showed that some gated linear unit (GLU) variants in the feed-forward sublayers of a transformer improve quality over ReLU and GELU. The gradient must pass through this rule, so it is chosen to be differentiable almost everywhere.
Example
Developers move a document recognition model to a new server and find that its rare activation function is not accelerated there. They replace it with the common ReLU, fine-tune the model, and the processing speed comes back.
Related terms
Questions and answers
Sources
- Nair, Hinton, Rectified Linear Units Improve Restricted Boltzmann Machines, ICML 2010cs.toronto.edu
- Glorot, Bordes, Bengio, Deep Sparse Rectifier Neural Networks, AISTATS 2011proceedings.mlr.press
- Hendrycks, Gimpel, Gaussian Error Linear Units (GELUs), 2016arxiv.org
- Shazeer, GLU Variants Improve Transformer, 2020arxiv.org
- Rumelhart, Hinton, Williams, Learning representations by back-propagating errors, Nature 323, 09.10.1986cs.toronto.edu
Not sure whether your company needs this?
AI solutions map