A team about to launch a customer support agent asks colleagues to try to break it: talk it into revealing another customer’s order, get it to issue a refund it should refuse, or slip instructions into a pasted document. Each failure found this way is one the team can fix before a real user finds it.
Microsoft’s AI Red Team drew on more than 100 generative AI products it had red teamed for a 2025 paper of lessons learned. Three of its eight lessons bear on this: AI red teaming is not safety benchmarking, automation can help cover more of the risk landscape, and the human element of the work is crucial.
Common targets:
- harmful or policy-violating outputs, such as unsafe instructions
- prompt injection, where instructions hidden in a document, web page or tool result redirect the system
- data exposure, such as revealing personal data, secrets or another user’s records
- unsafe actions by agents, such as calling a tool or changing a record without the right permission
How is AI red teaming different from AI model evaluation?
- AI model evaluation measures how well a system performs on a defined set of tasks, usually with fixed test cases and a score.
- AI red teaming searches for failures nobody has written a test case for yet, often with people actively trying to make the system misbehave.
- Confirmed red-team findings can become new test cases, so the two practices feed each other. See model evaluation, agent evaluation and LLM evaluation.
Where does AI red teaming break in AI work?
AI red teaming breaks when the result is reported as a count. A run that generates thousands of attack prompts can return a long list of failures, and a long list adds review work: someone still has to work out which failures are distinct, which conditions were never tried, and which findings need a decision from the team that owns the system.
A more useful red-team report groups findings by the condition that caused them, such as a type of request, a permission or a data source. The report also states which conditions remain untested and links each confirmed failure to the test case or guardrail change that addresses it, so the cases can be rerun after a model or prompt change to see whether the fix held.