MMOOZZGG.com

Glossary

Regularization in machine learning

In short

Regularization is a set of techniques that stop a model from fitting random details of its training data, so that it works better on new data.

Machine learning · regularization

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you are connecting dots in a picture. You can draw the line exactly through every dot, and you get a funny zigzag scribble. Or you can draw a smooth line close to the dots, and the picture becomes clearer. The teacher says: “Don’t jerk the pencil.” Adults made up a similar rule for computers, and it is called regularization.

In plain words

A model with many settings can fit any examples, even random typos in them. Regularization gets in the way of that. A simple method is to add a penalty for very large weights to the error, so that tangled explanations do not pay off for the model. Other methods switch off some neurons at random during training, stop training at the right moment, or create extra examples artificially. The price is that the model misses slightly more often on training data, but it holds up better on new data. In the same way, an experienced accountant does not build a forecast on one good month but looks at the whole year.

For business

These techniques decide whether a model from a demo will work the same way after launch. Constraints that are too weak lead to overfitting and nice numbers on old data. Constraints that are too strong make the model crude and deaf to important signals. The strength is tuned on held-out data, and these are separate training runs that belong in the budget. A useful side effect: L1 regularization sets extra features to zero, so the model is easier to explain to reviewers and auditors. Ask the contractor which methods they used, how they tuned the strength and on what data they tested it. Without a clear answer, the numbers in the presentation mean little. These questions are part of checking an AI project before you buy it.

For a specialist

Regularization covers changes to training that lower the generalization error at the cost of a possible rise in the training error. The explicit form adds a penalty to the loss function: L2 (ridge regression) penalizes the squared norm of the weights, and L1 (lasso) penalizes the sum of their absolute values and sets some coefficients to zero; Tibshirani proposed the lasso (JRSS B, 1996). Elastic Net combines both norms, and the alpha coefficient in scikit-learn sets the shrinkage strength. In neural networks, L2 is usually applied as weight decay; Loshchilov and Hutter (ICLR 2019) showed that for Adam these are not the same and proposed decoupled weight decay (AdamW). Methods without a penalty term include dropout (Srivastava and co-authors, JMLR, 2014), early stopping on a validation metric, data augmentation and label smoothing. The regularization strength is a hyperparameter, tuned on a validation set or by cross-validation.

Example

A bank built a model of responses to a loan offer from three hundred fields of the application form. Without a penalty, the model clung to random fields and fell apart on the next month’s data. With L1 regularization about twenty features remained, and the forecast became stable.

Related terms

Questions and answers

Why is regularization needed? It helps a model learn general patterns instead of memorizing the noise of specific examples. Without it, a flexible model often works brilliantly on training data and poorly on new data.
How do L1 and L2 regularization differ? L2 shrinks all weights in proportion to their size but almost never sets them to zero. L1 sets some weights to zero and so selects features.

You can combine them; this variant is called Elastic Net.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us