Glossary
Batch size in machine learning: what a batch is
In short
A batch is a group of training examples from which a model calculates its error and updates its weights in one step. Batch size in machine learning is the number of examples in that group.
Machine learning · batch
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you need to carry a whole mountain of toy blocks into your room. Carrying one block at a time takes too long, and you cannot lift the whole mountain at once. So you take a handful, carry it over and come back for the next one. A computer also learns from handfuls of examples: it looks at a small pile, corrects itself a little and takes a new one. Such a handful is called a batch.
In plain words
There can be millions of training examples, and you cannot show them all to the model at once: there is not enough memory. Showing them one by one is also bad, because training takes a long time and jumps around. So the data is cut into groups of equal size. The model looks at a group, works out where it made mistakes on average and makes one correction. Then it takes the next group. The group size is chosen to fit the GPU memory and the task. It is like grading homework: a teacher takes a stack, finds the common mistakes and explains them to the whole class instead of going through each paper alone.
For business
Batch size decides what hardware you need to train or fine-tune a model and how long it will take. A large batch needs more video memory, but it keeps the GPUs fully busy and cuts the number of steps. In a 2017 study, an image recognition model was trained in 1 hour on 256 GPUs with a batch of 8,192 images and no loss of accuracy, but the learning rate had to change by a special rule. The practical lesson for a company: if a contractor suggests buying a server for fine-tuning, ask which batch size they assumed and whether they compared quality with smaller ones. Sometimes a modest GPU with small batches gives the same result for less money, just more slowly.
For a specialist
A batch (mini-batch) is a subset of the training set used to estimate the gradient of the loss function in one optimization step. The batch size sets a trade-off: small batches give a noisy gradient and more steps per epoch, while large ones estimate the gradient more accurately and load the GPU more fully, but need more memory. Keskar and coauthors (ICLR 2017) showed that training with large batches tends to converge to sharp minima and generalizes worse, while small batches find flat minima thanks to their noise. Masters and Luschi (2018) got their top results with mini-batch sizes from 2 to 32. Goyal and coauthors (2017) kept accuracy at a size of 8,192 thanks to linear scaling of the learning rate and warmup. If the batch you need does not fit in memory, use gradient accumulation. The statistics of batch normalization, which Ioffe and Szegedy described in 2015, also depend on this size.
Do not confuse. Not to be confused with batch processing of requests (batch API), when a ready model answers a set of stored requests at once instead of learning.
Example
A team fine-tunes a model that sorts customer requests on a server with one GPU. A batch of 64 examples does not fit in memory, so the team takes 16 and accumulates gradients over four steps: the weights update almost as they would with a batch of 64, and the memory is enough.
Related terms
Questions and answers
Sources
- Keskar and coauthors, On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, ICLR 2017arxiv.org
- Masters, Luschi, Revisiting Small Batch Training for Deep Neural Networks, 2018arxiv.org
- Goyal and coauthors, Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017arxiv.org
- Ioffe, Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, 2015arxiv.org
Not sure whether your company needs this?
AI solutions map