Glossary
What is an NPU?
In short
An NPU (neural processing unit) is a block inside the chip of a phone or laptop that runs neural networks right on the device and uses little energy for it.
Infrastructure · neural processing unit
MMOOZZGG.com editorsChecked: 1 min read
For a child
There is a sharpener in your pencil case. It can do only one job, but it does it fast and does not take strength from your pens and pencils. Your phone has a small helper like that too. It handles the smart tricks: it recognizes a face in a photo, translates words and removes noise when you talk to your grandmother. And the battery runs down more slowly.
In plain words
A modern smartphone or laptop has several computing blocks. The central processor runs the whole system, the graphics processor draws the picture, and the neural processing unit takes on neural networks. It multiplies low-precision numbers fast and hardly heats up. So background blur on a video call, speech recognition and photo search work without the internet and do not drain the battery. For example, you dictate a message in the subway with no signal, and the phone still turns your voice into text: the model runs on the device itself.
For business
For a company, this block is useful where data should not go to the cloud: on employees’ laptops, in terminals, cameras and checkout systems. A model on the device answers without network delay, works offline and creates no token costs. There are limits too: the block is built for small compressed models, and a language model at the level of cloud services will not fit. Vendors set hardware requirements in operations per second: for Copilot+ PC features, Microsoft requires an NPU of at least 40 trillion operations per second (TOPS). Before you buy a fleet of devices, check whether the software you need supports this accelerator and what speed it gives on your tasks; a pilot measures this.
For a specialist
A neural processing unit (NPU) is a specialized accelerator for deep learning operations, built into a system on a chip (SoC) next to the CPU and GPU. In hardware, it is arrays of multiply-accumulate units tuned for low-bit integer arithmetic. As Microsoft notes in its developer guide for Copilot+ PCs, many such devices support only integer computation such as INT8, so FP32 models are quantized first. An application does not call the block directly but goes through a runtime and execution providers. On Windows this is Windows ML: it works on top of ONNX Runtime, finds the accelerator by itself and falls back to the GPU or CPU on failure. Mobile chips from various makers have similar blocks. Performance is given in TOPS, but real speed depends on memory bandwidth, support for the operators of a given model and the quality of the compiler.
Example
A service company gives its engineers laptops with a neural processing unit. Transcribing voice notes from a site and searching manuals work on the laptop itself, even in a basement with no signal, and the recordings never go to an outside service.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map