Glossary
What is a tensor processing unit?
In short
A tensor processing unit (TPU) is a specialized Google chip that speeds up training and running of neural networks. It does this by multiplying matrices very fast.
Infrastructure · tensor processing unit
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine a class with one student who can do everything: draw, sing, solve problems. Another student can only add numbers in columns, but does it faster than anyone in the school. If you need to add a million numbers, you call the second one. Smart assistants keep adding and multiplying huge tables of numbers. This chip was made for them, like that second student: it knows one job and does it very fast.
In plain words
An ordinary computer processor is universal: it opens files, shows video and calculates spreadsheets. A graphics card solves many identical tasks at once. The tensor processing unit goes further: it was designed only for neural network math. Almost all of a model’s work comes down to multiplying large tables of numbers, and this chip has a separate block that multiplies them whole, like an assembly line. Google puts these chips in its data centers and rents out their power in the cloud. For example, a developer trains a model that recognizes goods in a warehouse and pays for the chips’ time by the hour.
For business
For most companies, a TPU is not a hardware purchase. It is one way to rent compute in Google Cloud, next to graphics cards. It pays off on long training runs of large models and on recommendation systems with huge embedding tables. Google’s own documentation calls tasks with frequent branching, custom operations or high-precision math a poor fit. The price depends on the number of chips, the reservation term and the region. The risks: code and models get tied to one provider’s cloud, and data is processed outside Russia, which runs into Russia’s rules on personal data. So before choosing, companies compare cloud accelerators, GPU rental in Russia and their own server; this comparison is part of the AI solution map.
For a specialist
A Tensor Processing Unit is an application-specific integrated circuit (ASIC) that Google developed to speed up machine learning. Jouppi et al. (2017) described the first generation. The chip has run in the company’s data centers since 2015. Its matrix unit holds 65,536 8-bit multiply-accumulators (MAC) and gives a peak of 92 trillion operations per second (TOPS). On inference, it was 15–30 times faster than the CPUs and GPUs of its time and 30–80 times more efficient in operations per watt. The core of the architecture is a systolic array in the MXU block. In the current Cloud TPU documentation, it is a 128 by 128 array, so tensor dimensions are most efficient as multiples of 128. XLA (Accelerated Linear Algebra) compiles programs for the chip, and chips are joined into slices for distributed training. Models with dynamic tensor shapes and custom operations in the training loop are a poor fit.
Example
A startup trains a recommendation model for a marketplace on billions of views. Instead of buying servers, the team rents a TPU slice for the training period and runs the finished model on ordinary graphics cards.
Related terms
Questions and answers
How is a TPU different from a GPU?
How do you get access to a TPU?
Sources
Not sure whether your company needs this?
AI solutions map