Glossary
What is a tensor in machine learning?
In short
A tensor in machine learning is a multi-dimensional table of numbers where neural networks store input data, weights and middle results; a single number, a vector and a matrix are special cases of it.
Math of AI · tensor
MMOOZZGG.com editorsChecked: 1 min read
For a child
One number is like one sweet. A row of numbers is like sweets laid out in a line. A table of numbers is like a box with sweets in rows. And if you stack several such boxes, you get a big chest with floors. A computer keeps numbers in chests of different shapes like these, and they are called tensors.
In plain words
A neural network works with a huge amount of numbers, and it is easier to keep them not loose but in neat multi-dimensional tables. Such a table has a shape: how many dimensions it has and how long each one is. A color photo, for example, is height, width and three color channels, that is, three layers of numbers. A batch of a hundred photos adds one more dimension. AI programs add and multiply such tables whole, with one command, and a graphics card does it fast because it computes thousands of numbers at once.
For business
For an executive this concept matters through the money spent on hardware. Model weights are stored in exactly these arrays, and their size and number precision decide how much video memory you need and which server to buy. If each number is stored in 16 bits instead of 32, the model takes about half the space; quantization to 8 bits and lower cuts the size even more but can lower answer quality. So when you choose your own server, count not only the number of parameters but also the number format, context length and request batch size. A mistake in the memory estimate is expensive; test the model on the target graphics card before you buy to avoid it.
For a specialist
In deep learning, a tensor is an array of numbers arranged on a regular grid with a variable number of axes; this is how Goodfellow, Bengio and Courville define it in the book Deep Learning (2016). The number of axes is called the rank or order (rank, ndim), and the set of lengths along the axes is the shape. A scalar has zero axes, a vector one, a matrix two. The PyTorch documentation describes torch.Tensor as a multi-dimensional matrix of elements of one data type: float32 by default, also float16, bfloat16, int8 and others; it sits in CPU or GPU (CUDA) memory. Typical shapes: an image batch (N, C, H, W), a sequence of embeddings (batch, seq_len, d_model). Element-wise operations, convolutions, matrix multiplication and broadcasting run in the library on the GPU, and automatic differentiation computes gradients over such arrays. In physics and differential geometry the concept is stricter: a multilinear object with rules for transformation when the basis changes.
Do not confuse. Do not confuse it with the tensor in physics and mechanics, a quantity with special rules for conversion when the coordinate system changes: in AI it is a multi-dimensional array of numbers.
Example
For example, a batch of 32 color product photos of 224 by 224 pixels enters the model as one tensor of shape (32, 3, 224, 224), and the model processes the whole batch in one pass.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map