Guide
AI red teaming is structured, adversarial testing of an AI system. Testers act as attackers to find security holes, safety failures, bias and unintended behavior before those problems reach real users.
Why it is different
Traditional security testing assumes software that behaves the same way every time. AI systems do not, and that changes how they have to be tested.
| Traditional security testing | AI red teaming | |
|---|---|---|
| Behavior | The same input gives the same output | The same input can give different outputs each time |
| Attack | Exploits written in code | Often plain language anyone can type |
| Attack surface | Fixed and mapped in advance | Shifts with context, data and conversation history |
| Scope | Security | Security, plus safety and fairness |
| Result | Exploited or not exploited | Often a judgment about whether an output is harmful |
Approaches
Human testers find new attacks and subtle harms that no script anticipates. It is the slowest and most expensive approach.
Tools run thousands of attack variations. Good for breadth and for retesting after changes, but they can miss flaws that depend on context.
People design the strategy and judge the results while tools run attacks at scale. Published guidance generally recommends this approach.
Black box
No inside knowledge. The tester sees what an outside attacker would see.
Gray box
Partial knowledge, such as the system prompt or tool list. Close to an insider's view.
White box
Full access to code, configuration and model. The most thorough option.
Defense layers
A deployed AI system has up to five layers of defense. A good red teamer first works out which layer blocked a request, then picks a technique aimed at that layer.
Each layer should be tested as if the others have already failed. Systems that guard only the entrance leave retrieved documents, memory and tool results unprotected.
Input filters
Classifiers that screen a request before the model sees it.
Model alignment
The refusals the model learned in training.
System prompt and identity
The persona and limits the operator wrote for the assistant.
Output filters
Checks on a response after the model has written it.
Agent trust boundaries
The controls around retrieval, tools, memory and routing between agents.
Who does it
When to do it
What gets measured
Attack success rate
The share of attack attempts that worked
Time to compromise
How long it took to break the system
Coverage
How much of the risk surface was tested
False positive rate
How many reported issues turned out not to be real
Severity distribution
The mix of critical, high, medium and low findings over time
Standards and rules that call for it
NIST AI Risk Management Framework
Includes red teaming as part of testing AI systems
EU AI Act, Article 15
High-risk systems must show accuracy, robustness and cybersecurity
OWASP GenAI Red Teaming Guide
Practical guidance, alongside the OWASP Top 10 lists for LLM and agentic applications
MITRE ATLAS
A shared catalogue of attacker tactics for planning coverage
Pro Red Team is still being finalized, and services are not yet available for purchase. To see how we apply these ideas, read our methodology.
Read the methodology