Glossary
Regularization in machine learning
In short
Regularization is a set of techniques that stop a model from fitting random details of its training data, so that it works better on new data.
Machine learning · regularization
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine you are connecting dots in a picture. You can draw the line exactly through every dot, and you get a funny zigzag scribble. Or you can draw a smooth line close to the dots, and the picture becomes clearer. The teacher says: “Don’t jerk the pencil.” Adults made up a similar rule for computers, and it is called regularization.
In plain words
A model with many settings can fit any examples, even random typos in them. Regularization gets in the way of that. A simple method is to add a penalty for very large weights to the error, so that tangled explanations do not pay off for the model. Other methods switch off some neurons at random during training, stop training at the right moment, or create extra examples artificially. The price is that the model misses slightly more often on training data, but it holds up better on new data. In the same way, an experienced accountant does not build a forecast on one good month but looks at the whole year.
For business
These techniques decide whether a model from a demo will work the same way after launch. Constraints that are too weak lead to overfitting and nice numbers on old data. Constraints that are too strong make the model crude and deaf to important signals. The strength is tuned on held-out data, and these are separate training runs that belong in the budget. A useful side effect: L1 regularization sets extra features to zero, so the model is easier to explain to reviewers and auditors. Ask the contractor which methods they used, how they tuned the strength and on what data they tested it. Without a clear answer, the numbers in the presentation mean little. These questions are part of checking an AI project before you buy it.
For a specialist
Regularization covers changes to training that lower the generalization error at the cost of a possible rise in the training error. The explicit form adds a penalty to the loss function: L2 (ridge regression) penalizes the squared norm of the weights, and L1 (lasso) penalizes the sum of their absolute values and sets some coefficients to zero; Tibshirani proposed the lasso (JRSS B, 1996). Elastic Net combines both norms, and the alpha coefficient in scikit-learn sets the shrinkage strength. In neural networks, L2 is usually applied as weight decay; Loshchilov and Hutter (ICLR 2019) showed that for Adam these are not the same and proposed decoupled weight decay (AdamW). Methods without a penalty term include dropout (Srivastava and co-authors, JMLR, 2014), early stopping on a validation metric, data augmentation and label smoothing. The regularization strength is a hyperparameter, tuned on a validation set or by cross-validation.
Example
A bank built a model of responses to a loan offer from three hundred fields of the application form. Without a penalty, the model clung to random fields and fell apart on the next month’s data. With L1 regularization about twenty features remained, and the forecast became stable.
Related terms
Questions and answers
How do L1 and L2 regularization differ?
Sources
- Tibshirani, Regression Shrinkage and Selection via the Lasso, JRSS B, 1996academic.oup.com
- scikit-learn, Linear Models: Ridge, Lasso, Elastic-Net, documentationscikit-learn.org
- Loshchilov, Hutter, Decoupled Weight Decay Regularization, ICLR 2019arxiv.org
- Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR, 2014jmlr.org
Where companies use it
Not sure whether your company needs this?
AI solutions map