MMOOZZGG.com

Glossary

What is a tensor processing unit?

In short

A tensor processing unit (TPU) is a specialized Google chip that speeds up training and running of neural networks. It does this by multiplying matrices very fast.

Infrastructure · tensor processing unit

MMOOZZGG.com editorsChecked: 2 min read

For a child

Imagine a class with one student who can do everything: draw, sing, solve problems. Another student can only add numbers in columns, but does it faster than anyone in the school. If you need to add a million numbers, you call the second one. Smart assistants keep adding and multiplying huge tables of numbers. This chip was made for them, like that second student: it knows one job and does it very fast.

In plain words

An ordinary computer processor is universal: it opens files, shows video and calculates spreadsheets. A graphics card solves many identical tasks at once. The tensor processing unit goes further: it was designed only for neural network math. Almost all of a model’s work comes down to multiplying large tables of numbers, and this chip has a separate block that multiplies them whole, like an assembly line. Google puts these chips in its data centers and rents out their power in the cloud. For example, a developer trains a model that recognizes goods in a warehouse and pays for the chips’ time by the hour.

For business

For most companies, a TPU is not a hardware purchase. It is one way to rent compute in Google Cloud, next to graphics cards. It pays off on long training runs of large models and on recommendation systems with huge embedding tables. Google’s own documentation calls tasks with frequent branching, custom operations or high-precision math a poor fit. The price depends on the number of chips, the reservation term and the region. The risks: code and models get tied to one provider’s cloud, and data is processed outside Russia, which runs into Russia’s rules on personal data. So before choosing, companies compare cloud accelerators, GPU rental in Russia and their own server; this comparison is part of the AI solution map.

For a specialist

A Tensor Processing Unit is an application-specific integrated circuit (ASIC) that Google developed to speed up machine learning. Jouppi et al. (2017) described the first generation. The chip has run in the company’s data centers since 2015. Its matrix unit holds 65,536 8-bit multiply-accumulators (MAC) and gives a peak of 92 trillion operations per second (TOPS). On inference, it was 15–30 times faster than the CPUs and GPUs of its time and 30–80 times more efficient in operations per watt. The core of the architecture is a systolic array in the MXU block. In the current Cloud TPU documentation, it is a 128 by 128 array, so tensor dimensions are most efficient as multiples of 128. XLA (Accelerated Linear Algebra) compiles programs for the chip, and chips are joined into slices for distributed training. Models with dynamic tensor shapes and custom operations in the training loop are a poor fit.

Example

A startup trains a recommendation model for a marketplace on billions of views. Instead of buying servers, the team rents a TPU slice for the training period and runs the finished model on ordinary graphics cards.

Related terms

Questions and answers

How is a TPU different from a GPU? A graphics card is more universal: it fits graphics, scientific computing and almost any neural network, and more software is written for it. A tensor processing unit is built for matrix multiplication and wins on large, uniform tasks.

Server versions of these chips are available through Google’s cloud services.

How do you get access to a TPU? Through Google Cloud: the documentation names Compute Engine, Google Kubernetes Engine and the company’s platform for agents. Your code has to be compiled with the XLA compiler.

For work inside their own perimeter, companies usually choose servers with graphics cards.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us