MMOOZZGG.com

Glossary

What is AI red teaming

In short

AI red teaming (adversarial testing) is a check in which a separate team looks on purpose for weak points in an AI system, before attackers or customers find them.

Security and law · red teaming

MMOOZZGG.com editorsChecked: 1 min read

For a child

Before children can play in a new playground, adults climb the slides themselves, shake the railings and look for places where a jacket could get caught. They do not break the playground; they make sure nobody gets hurt. People do the same with smart programs: special testers look in advance for places where a program could make a mistake or say something bad, and these get fixed before anyone starts using it.

In plain words

The name comes from military exercises, where the “red team” plays the enemy. In AI it means a test in which people try on purpose to make a model or a chatbot behave wrongly: reveal someone else’s data, be rude, give dangerous advice, break company rules. Each problem found is recorded, the developers fix it, and then the test runs again. A regular test checks whether the system works when people use it as intended. Here the test looks at the opposite: what happens when someone acts against the rules.

For business

Companies run such a test before an AI agent goes live with customers and after big changes: a new model, new tools, access to the CRM. The goal is simple: learn about the risks before they show up in reviews. The test covers leaks of personal data and business information, ways around the limits, reactions to provocation, and false promises of discounts and dates. The cost depends on the number of scenarios, how deep the agent’s access to systems goes, and who tests: your own team, outside experts or automatic test generators. The result is a list of problems with priorities, and the launch waits until the critical ones are fixed. In a pilot, this test runs on the real scenarios of your process.

For a specialist

The NIST AI 600–1 profile for generative AI (July 2024) defines AI red teaming as a structured exercise that looks for flaws and weaknesses of a system: inaccurate, harmful or discriminatory answers. The profile separates tests by the general public, by experts in the subject area, by mixed teams and by pairs of a person and a generative model. Ganguli and co-authors (2022) described manual testing of models with 2.7, 13 and 52 billion parameters. They published 38,961 recorded attempts and showed that models trained with RLHF are harder to push toward harmful answers as they grow. Perez and co-authors (2022) proposed that another language model generates the test cases and a classifier rates the answers, which costs less than manual work. The process includes a threat model, a plan to cover the risks, testing sessions, labeling of findings, fixes (fine-tuning, guardrails, filters) and a repeat check. The result does not prove that a system is safe: it shows only the classes of failure that were found.

Example

Before an online store’s support agent goes live, the team tests it on risky scenarios: will it show someone else’s order, promise a discount outside the rules, stay on topic? The agent promised a discount that did not exist several times. A ban was added to its instructions, and the repeat check passed clean.

Related terms

Questions and answers

How is red teaming different from regular testing? A regular test checks that the system does what it should. Adversarial testing looks at what the system does when someone gets in its way or tries to trick it, so it finds other errors.
Who does red teaming? Your own team, outside experts or a wide group of users, often together with automatic test generators. Narrow fields, such as medicine or law, need experts from that field.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us