Glossary
What is CUDA?
In short
CUDA is a platform and programming model from NVIDIA. Through it, programs use a graphics card for parallel calculations, including training and running neural networks.
Infrastructure · CUDA
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a huge choir where hundreds of children sing at once. To make a song, they need a conductor and notes that everyone understands. A graphics card is like such a choir of thousands of tiny singers, but instead of singing they count. A special set of rules works like the notes and the conductor’s hands: with them, a program tells the choir who does which part of the work. Without notes, the choir stays silent.
In plain words
A graphics card can do thousands of calculations at the same time, but an ordinary program does not know this. It needs a way to say: split the work into pieces and compute them in parallel. NVIDIA released such a way for its graphics cards. A developer writes a function in a familiar language like C++, and it runs in thousands of threads at once. Ready libraries are built on top of this base, and popular neural network frameworks use them without the user noticing. That is why a program for a local neural network often asks for an NVIDIA graphics card and a driver with this platform.
For business
This platform explains why NVIDIA graphics cards are so common in AI: many libraries, engines and guides are written for it. Hardware without it narrows the list of programs that work right away and gives engineers porting work. This is also a form of vendor dependence: card prices, delivery times and import limits affect the project budget. There are alternatives: accelerators from other makers with their own software stacks, and Mac computers for small models. Before you buy, check whether your chosen inference engine supports your hardware and which driver version it needs. Choosing a server for specific models is part of an AI solution map.
For a specialist
NVIDIA introduced CUDA in November 2006 as a general-purpose parallel computing platform and programming model for its GPUs. The programmer writes kernels: C++ functions with the __global__ specifier that run N times in parallel in N threads when called. Threads are grouped into blocks of up to 1024, and blocks into a grid of one, two or three dimensions. Blocks must run independently and in any order, so one compiled program scales to any number of multiprocessors. Besides C++, the guide names Fortran, DirectCompute and OpenACC. For deep learning, the library layer matters. Chetlur and co-authors (2014) presented cuDNN, a set of optimized primitives modeled on BLAS; adding it to Caffe sped up a standard model by 36% and cut memory use. Frameworks usually rely on such libraries instead of writing each kernel from scratch.
Example
A company buys a server for a local model and chooses between two accelerators with the same memory. The engineer finds that the needed inference engine officially supports only one of them, and that settles the choice faster than a price comparison.
Related terms
Questions and answers
Do you need to know CUDA to run a neural network?
Sources
Not sure whether your company needs this?
AI solutions map