Glossary
What is a GPU?
In short
A GPU (graphics processing unit) is a graphics processor: a chip that performs thousands of identical calculations at the same time, so neural networks are trained and run on it.
Infrastructure · graphics processing unit
MMOOZZGG.com editorsChecked: 2 min read
For a child
Imagine you need to color a thousand identical pictures. One artist would take a very long time, but a whole class can finish in one lesson if each student takes a picture. A normal processor is like one smart artist. A GPU is like a whole class: each student does simple work, but all of them work together at the same time. Neural networks need exactly that: a lot of simple work at once.
In plain words
Graphics cards were invented for computer games. To draw a frame, the card has to compute the color of millions of points on the screen, and all these calculations are alike. Later it turned out that a neural network works the same way: its job comes down to a huge number of multiplications and additions that can run in parallel. So a graphics card computes a neural network much faster than a normal processor. When you ask a chatbot a question, a graphics card in a data center most likely computes the answer. And a small model can run on a home computer if its graphics card has enough memory.
For business
If a model runs on your own servers, the GPU becomes the main AI expense. The price depends first of all on video memory: it decides which model fits and how many requests the server handles at once. There are three paths: buy a server, rent graphics cards by the hour, or pay a cloud API per token. Buying pays off with a steady, high load; renting suits testing ideas and peaks; a cloud API suits the start. Before you buy, estimate the load in requests per day, the data storage rules, electricity, cooling and who will maintain the hardware. Make this estimate in a pilot with real requests, not from the card’s spec sheet.
For a specialist
A graphics processing unit is a massively parallel accelerator with the SIMT execution model (single instruction, multiple threads): threads run in groups (warps), each group executes one instruction on different data, and thousands of threads work on the chip at the same time. The NVIDIA CUDA guide says such a chip gives more transistors to data processing rather than to caching and flow control, and it beats the CPU in instruction and memory throughput. For neural networks this is decisive, because training and inference come down to matrix multiplication (GEMM) and convolutions. The turning point came in 2012: Krizhevsky, Sutskever and Hinton trained the AlexNet convolutional network with 60 million parameters on an efficient GPU implementation. Today, for LLMs, what matters is memory size and bandwidth (HBM or GDDR), performance in FP16, BF16, FP8 and INT8, and the speed of links between cards (NVLink, PCIe). Alternatives: TPUs, NPUs and other specialized accelerators (ASICs).
Example
An online shop runs an AI assistant on its own server. There are many requests by day and almost none at night, so during the pilot the company rents GPUs by the hour and buys a server only once the load is clear.
Related terms
Questions and answers
Which graphics card do you need for AI?
Sources
Not sure whether your company needs this?
AI solutions map