Glossary
What is FLOPS?
In short
FLOPS (floating point operations per second) is the number of floating point operations per second, a measure of the computing power of processors and graphics cards for neural networks.
Infrastructure · FLOPS
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine a mental math contest: who can solve more sums in a minute. One student solves ten, another thirty. Computers compete too, only they solve billions of sums every second. The number of sums solved per second shows how strong a machine is at counting. The bigger this number, the faster it trains and runs smart helpers.
In plain words
Inside a neural network, almost everything comes down to multiplying and adding decimal numbers. Each such action is called a floating point operation. Count how many a chip does per second, and you get its power. The figures are huge, so prefixes are used: a teraflops is a trillion operations per second, and a petaflops is a thousand times more. A similar value measures the amount of work: how many operations in total went into training a model. For example, graphics card makers list power in teraflops in their specifications, and reports on large models give the total computing spent on training.
For business
The amount of computing sets the training budget: hours of graphics card rental are multiplied by their power and their real load. For language models, there is a rule of thumb from the work of Kaplan and co-authors (2020): training needs about 6 operations for each parameter for each token of data, so the cost grows with both the model size and the corpus size. Peak power from the specification is not working power: chips are almost never fully loaded, and when you run a ready model, speed is more often limited by memory. The number of operations has also entered regulation: under the EU AI Act, a general-purpose model counts as having systemic risk if its training used more than 10 to the power of 25 operations. The computing estimate for your task is made before you rent or buy hardware, during the pilot.
For a specialist
FLOPS (floating point operations per second) measures speed, while FLOP, or FLOPs with a lower-case s, means the number of operations; the literature often mixes these up. The peak performance of an accelerator is given separately for the FP64, FP32, TF32, BF16, FP16 and FP8 formats, and the figures for low precision are much higher. Kaplan and co-authors (2020) estimate training compute as C ≈ 6N operations per token, where N is the number of non-embedding parameters, and express budgets in PF-days: 10^15 × 24 × 3600 ≈ 8.64 × 10^19 operations. Real hardware load in training is well below the peak, and it is compared as the share of useful model operations in the rated power. For LLM inference, arithmetic intensity matters more: decoding reads all the weights for every token and is limited by memory, not by computing. The European Commission links the systemic risk threshold of general-purpose models to 10^25 training operations and allows it to be revised.
Example
A lab plans to fine-tune an open model on its reports. An engineer estimates the amount of computing from the number of parameters and tokens, divides it by the real, not the rated, power of the rented cards, and gets the training time in hours, and with it the budget.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map