Glossary
What is AI safety
In short
AI safety is a field of research and practice that makes sure AI systems do not cause harm: they do not make dangerous mistakes, get out of control or serve attackers.
AI basics · AI safety
MMOOZZGG.com editorsChecked: 1 min read
For a child
When you ride a bike, you wear a helmet, and adults check the brakes in advance. The bike is great, but without these steps you can fall. Smart programs are the same: people check in advance that a program does not give harmful advice, does not mix up important things and listens to a person. This is what specialists in the safety of smart machines do.
In plain words
The more AI does on its own, the more important it is that it does exactly what was intended. AI safety studies how to achieve this. It covers different questions: how to stop a model from inventing dangerous advice, how to protect it from being tricked by attackers, how to make sure an AI agent does not delete extra files to complete a task. A separate topic is future powerful systems: how to keep control over them. It is like safety rules in a factory: they are written in advance, not after an accident.
For business
In a company, this work comes down to clear steps. An agent gets only the rights it needs: it may read the CRM but not delete deals. A person confirms important actions, such as a payment or a letter to a client. Guardrails check the model’s answers, and before launch the model itself is tested with provocative requests. All actions are written to a log so that an error can be analyzed later. Such measures usually cost less than one incident: a leak, a wrong payout, a complaint to a regulator. Which data employees may send to AI tools and which they may not is set in the AI use policy.
For a specialist
AI safety studies how to prevent harm from AI systems, from design errors to deliberate misuse. The program paper for machine learning was Concrete Problems in AI Safety by D. Amodei and co-authors (2016), with five problems: avoiding side effects and reward hacking, ensuring scalable supervision, safe exploration and robustness to distributional shift. The International AI Safety Report led by Y. Bengio, published on 3 February 2026, splits the risks of general-purpose AI into three groups: malicious use, malfunctions (including loss of control) and systemic risks. Practical frameworks: the voluntary NIST AI Risk Management Framework 1.0 (January 2023), evaluation of dangerous capabilities before a model is released, red teaming, interpretability and alignment. A neighboring field, AI security, deals with attacks: prompt injection, data poisoning and theft of model weights.
Example
A company connects an AI agent to email and the CRM. Before launch, the security team allows it only to read and to write drafts, and leaves sending letters to clients to the manager. A month later, the agent by mistake prepares a letter to the wrong client, but it stays a draft.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map