Glossary
What is AI red teaming
In short
AI red teaming (adversarial testing) is a check in which a separate team looks on purpose for weak points in an AI system, before attackers or customers find them.
Security and law · red teaming
MMOOZZGG.com editorsChecked: 1 min read
For a child
Before children can play in a new playground, adults climb the slides themselves, shake the railings and look for places where a jacket could get caught. They do not break the playground; they make sure nobody gets hurt. People do the same with smart programs: special testers look in advance for places where a program could make a mistake or say something bad, and these get fixed before anyone starts using it.
In plain words
The name comes from military exercises, where the “red team” plays the enemy. In AI it means a test in which people try on purpose to make a model or a chatbot behave wrongly: reveal someone else’s data, be rude, give dangerous advice, break company rules. Each problem found is recorded, the developers fix it, and then the test runs again. A regular test checks whether the system works when people use it as intended. Here the test looks at the opposite: what happens when someone acts against the rules.
For business
Companies run such a test before an AI agent goes live with customers and after big changes: a new model, new tools, access to the CRM. The goal is simple: learn about the risks before they show up in reviews. The test covers leaks of personal data and business information, ways around the limits, reactions to provocation, and false promises of discounts and dates. The cost depends on the number of scenarios, how deep the agent’s access to systems goes, and who tests: your own team, outside experts or automatic test generators. The result is a list of problems with priorities, and the launch waits until the critical ones are fixed. In a pilot, this test runs on the real scenarios of your process.
For a specialist
The NIST AI 600–1 profile for generative AI (July 2024) defines AI red teaming as a structured exercise that looks for flaws and weaknesses of a system: inaccurate, harmful or discriminatory answers. The profile separates tests by the general public, by experts in the subject area, by mixed teams and by pairs of a person and a generative model. Ganguli and co-authors (2022) described manual testing of models with 2.7, 13 and 52 billion parameters. They published 38,961 recorded attempts and showed that models trained with RLHF are harder to push toward harmful answers as they grow. Perez and co-authors (2022) proposed that another language model generates the test cases and a classifier rates the answers, which costs less than manual work. The process includes a threat model, a plan to cover the risks, testing sessions, labeling of findings, fixes (fine-tuning, guardrails, filters) and a repeat check. The result does not prove that a system is safe: it shows only the classes of failure that were found.
Example
Before an online store’s support agent goes live, the team tests it on risky scenarios: will it show someone else’s order, promise a discount outside the rules, stay on topic? The agent promised a discount that did not exist several times. A ban was added to its instructions, and the repeat check passed clean.
Related terms
Questions and answers
Sources
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024nvlpubs.nist.gov
- Ganguli et al., Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned, 2022arxiv.org
- Perez et al., Red Teaming Language Models with Language Models, 2022arxiv.org
Where companies use it
Not sure whether your company needs this?
AI solutions map