Glossary
What is dropout in a neural network?
In short
Dropout is a technique for training neural networks in which a share of neurons is switched off at random at every step, so the network does not rely on single connections.
Machine learning · dropout
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a football team where at every practice the coach puts a few random players on the bench. The others have to play without them, and each one learns to pass, to shoot and to defend. On match day everyone plays, and the team works well together. A computer is taught in a similar way, and this way is called dropout.
In plain words
A neural network is made of many small calculators, neurons. During training some of them start to work only in pairs with their neighbors and together memorize random details of the examples. To prevent this, at every step a share of neurons is switched off for a while, a different share each time. The network has to spread its knowledge so it can cope without any one of them. When training is over, nobody needs to be switched off: all neurons work at once. It is like a department where staff cover for each other during holidays and so know their colleagues’ work.
For business
Dropout costs almost nothing: one line in the model code and one number, the share of neurons to switch off. In return, it lowers the risk that a network fine-tuned on a small set of your documents or images memorizes the examples and starts making mistakes on new ones. It does not affect the speed or the cost of the finished model’s answers, because it is off in use. The catch is the setting: too large a share leaves the model undertrained, and too small a share does nothing. It is chosen by quality on held-out data. If a contractor fine-tunes a network on your data, ask which measures against overfitting they used, and ask them to compare the error on training and on validation. This comparison is part of the pilot report.
For a specialist
Dropout is a stochastic regularization of neural networks: at each training step the output of each neuron is set to zero independently with probability p. Hinton and colleagues proposed the method in a 2012 preprint, where half of the feature detectors were switched off on each training example to suppress their co-adaptation; Srivastava, Hinton and colleagues gave a full description and experiments (JMLR, 2014). Training can be seen as an approximate averaging of many thinned subnetworks with shared weights, while at test time one network with scaled-down weights works. PyTorch implements inverted dropout: during training the outputs are multiplied by 1/(1-p), and in evaluation mode the layer works as the identity function; p is 0.5 by default. Related variants: DropConnect, where single weights are set to zero, spatial dropout for convolutional maps, and Monte Carlo dropout, where the masks stay on at inference to estimate the uncertainty of predictions.
Example
A clinic fine-tuned a network to find fractures on X-ray images, and it had only a few thousand examples. Without dropout, accuracy on validation fell after a few epochs, although it grew on training. With a dropout share of about one third, the gap shrank, and the model worked more steadily on images from a different machine.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map