proredteam

Guide

AI red teaming, explained.

AI red teaming is structured, adversarial testing of an AI system. Testers act as attackers to find security holes, safety failures, bias and unintended behavior before those problems reach real users.

Why it is different

A penetration test is not enough for an AI system.

Traditional security testing assumes software that behaves the same way every time. AI systems do not, and that changes how they have to be tested.

Traditional security testing AI red teaming
Behavior The same input gives the same output The same input can give different outputs each time
Attack Exploits written in code Often plain language anyone can type
Attack surface Fixed and mapped in advance Shifts with context, data and conversation history
Scope Security Security, plus safety and fairness
Result Exploited or not exploited Often a judgment about whether an output is harmful

Approaches

Three ways to run the testing.

Manual

Human testers find new attacks and subtle harms that no script anticipates. It is the slowest and most expensive approach.

Automated

Tools run thousands of attack variations. Good for breadth and for retesting after changes, but they can miss flaws that depend on context.

Hybrid

How we work

People design the strategy and judge the results while tools run attacks at scale. Published guidance generally recommends this approach.

How much the tester knows

Black box

No inside knowledge. The tester sees what an outside attacker would see.

Gray box

Partial knowledge, such as the system prompt or tool list. Close to an insider's view.

White box

Full access to code, configuration and model. The most thorough option.

Defense layers

Where an AI system defends itself.

A deployed AI system has up to five layers of defense. A good red teamer first works out which layer blocked a request, then picks a technique aimed at that layer.

Each layer should be tested as if the others have already failed. Systems that guard only the entrance leave retrieved documents, memory and tool results unprotected.

  1. 01

    Input filters

    Classifiers that screen a request before the model sees it.

  2. 02

    Model alignment

    The refusals the model learned in training.

  3. 03

    System prompt and identity

    The persona and limits the operator wrote for the assistant.

  4. 04

    Output filters

    Checks on a response after the model has written it.

  5. 05

    Agent trust boundaries

    The controls around retrieval, tools, memory and routing between agents.

Who does it

A red team needs more than one kind of expert.

  • Security researchers with adversarial testing experience
  • Machine learning engineers who know how models are built and deployed
  • Domain experts in fields such as medicine, law and finance
  • Ethicists and social scientists for fairness and bias
  • Creative thinkers who can imagine misuse nobody has tried yet

When to do it

Testing is continuous, not a one-time event.

  • Full testing before each launch or major model update
  • Automated scans every week, or on every build
  • A deep-dive exercise every quarter
  • Extra testing when a new attack is published or the system changes
  • A retest after every fix

What gets measured

Five numbers show whether a system is getting safer.

Attack success rate

The share of attack attempts that worked

Time to compromise

How long it took to break the system

Coverage

How much of the risk surface was tested

False positive rate

How many reported issues turned out not to be real

Severity distribution

The mix of critical, high, medium and low findings over time

Standards and rules that call for it

NIST AI Risk Management Framework

Includes red teaming as part of testing AI systems

EU AI Act, Article 15

High-risk systems must show accuracy, robustness and cybersecurity

OWASP GenAI Red Teaming Guide

Practical guidance, alongside the OWASP Top 10 lists for LLM and agentic applications

MITRE ATLAS

A shared catalogue of attacker tactics for planning coverage

Demos are coming soon.

Pro Red Team is still being finalized, and services are not yet available for purchase. To see how we apply these ideas, read our methodology.

Read the methodology