Red teaming
In short
Red teaming is a test that probes a language model for harmful outputs, using people or another model, and then uses what it finds to measure and reduce those outputs.
- Also called
- AI red teaming, adversarial testing
- Topic
- Testing and benchmarks
- Level
- Some background helps
Ganguli and colleagues describe red teaming as using manual or automated methods to probe a language model for harmful outputs, then updating the model so it avoids those outputs1. They list harmful behaviour such models can show, including offensive or toxic text, reinforcing social biases, and leaking personal information from training data1.
People can run the probes, or another language model can write them. Perez and colleagues generated test questions with one model and judged the target model's replies with a classifier trained to detect offensive content2. That search found tens of thousands of offensive replies from a chatbot with 280 billion parameters2. Other prompts surfaced groups the chatbot discussed in offensive ways, personal and hospital phone numbers given as its own contact details, and private training data repeated in the output2.
An Anthropic study then had people attack models in open-ended conversation, at three sizes: 2.7 billion, 13 billion and 52 billion parameters1. The team released 38,961 attacks1. Recorded harms ran from offensive language to subtler non-violent unethical outputs, and a cluster of 916 attacks was designed to solicit personal information1. Safety changes they tested, including reinforcement learning from human feedback, still left harmful outputs. The authors say those changes did not make a perfectly safe system1.
A clean run is narrow too. If no test case draws a harmful reply, Perez and colleagues treat that as evidence of low risk only on the distribution of cases that were tested2.
Example
Testers hold an open-ended conversation with a support chatbot and try to make it say something harmful or reveal personal information. That is the task Ganguli and colleagues set their human red team1.
Why it matters
Language models can harm users in ways that are hard to predict before deployment2. Red teaming is one way to find some of those outputs and to check whether a safety change reduced them1.
Questions people ask
Who carries out AI red teaming?
People can do it, as in Anthropic's study of conversational attacks on models of three sizes1. Another language model can also write the test questions, with a classifier then checking replies for offensive content2.
Does a clean red-team result mean the model is safe?
No. If none of the tested cases draws a harmful reply, that is evidence of low risk only for the distribution of cases that were tested2. In the Anthropic study, the safety changes still left harmful outputs1.
What harms have these studies looked for?
Perez and colleagues looked for offensive replies, groups discussed offensively, leaked private training data, and phone numbers offered as the chatbot's own contact details2. Ganguli and colleagues recorded offensive language, non-violent unethical outputs, and attempts to obtain personal information1.