MMOOZZGG.com

Glossary

What are hyperparameters?

In short

Hyperparameters are training settings that a person sets before the start and that the model does not change itself: the learning rate, the batch size, the number of epochs and the size of the network.

Machine learning · hyperparameters

MMOOZZGG.com editorsChecked: 1 min read

For a child

When you bake a pie with your mom, you decide in advance which pan to use, how hot to make the oven and how many minutes to wait. The pie in the oven does not decide this. If you choose badly, it burns or stays raw. Before a computer learns, there are decisions like these too, and people make them. They are called hyperparameters.

In plain words

A model has two kinds of numbers. The first it finds itself during training: these are its weights. The second a person sets in advance, and they control the process itself: how fast to learn, how many examples to take at a time, how many passes to make over the data and how many layers the network will have. It is hard to guess them on the first try, so several options are tried and the results are compared on validation data. It is like a soup recipe: the ingredients are the same, but the taste depends on how long you cook it and when you add salt.

For business

Hyperparameter tuning takes a large part of the training budget: each set of settings means a separate run on graphics cards. If a contractor tries twenty combinations, you pay for twenty trainings, although only one goes into use. So ask in advance how many runs the estimate includes, how they are chosen and on which data they are compared. There is also a quality risk: if the settings were fitted by looking at the test set many times, the model will do worse on real clients than promised. Require a held-out set that the team did not touch until the final check, and recorded settings, so training can be repeated without the original team. Such terms are easy to fix in the pilot specification.

For a specialist

Hyperparameters are parameters of the training process and architecture that are not optimized by the gradient of the loss function: the learning rate, batch size, number of epochs, regularization coefficients, dropout rate, number and width of layers, and optimizer parameters. In scikit-learn terms, they are the arguments of an estimator’s constructor, such as max_depth of a random forest. Hyperparameter optimization is guided by quality on a validation set or by cross-validation. Grid search scales poorly to a high-dimensional space: Bergstra and Bengio (JMLR, 2012) showed that random search finds models as good as the grid at a small fraction of the computing. The next step is Bayesian optimization: Snoek, Larochelle and Adams (2012) modeled quality with a Gaussian process and chose the next point from its posterior distribution. A separate group of methods drops weak runs early (successive halving, Hyperband). The final score is taken on a test set that did not take part in tuning.

Example

A retail chain ordered a customer churn forecast model. The contractor tried 40 combinations of tree depth and learning rate and chose the option that gave fewer false alarms on held-out data for the last quarter.

Related terms

Questions and answers

How do hyperparameters differ from model parameters? Parameters, or weights, are found by the model itself during training. Hyperparameters are set by a person before the start, and they decide how training goes and what model you get.
How is hyperparameter tuning done? You try several sets of settings and compare quality on validation data. Instead of a full grid search, random search or Bayesian optimization is usually used: they find a good option in fewer runs.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us