AI Red Teaming: What Enterprise Leaders Should Actually Ask For

Two years ago, "red team the AI" was mostly what safety researchers did to unreleased models. Today it is a line item in enterprise AI budgets, a specific EU AI Act obligation for the largest general-purpose model providers, and — for most large enterprises deploying LLM-integrated workloads — a control the CISO is being asked about by insurers, auditors, and customers. This guide sets out what AI red teaming is, how it differs from adjacent activities, what the regulatory and standards landscape now requires, and what enterprise leaders should ask for when they commission or evaluate a programme.

What AI Red Teaming Is (and Is Not)

AI red teaming is a structured, adversarial testing effort to find flaws and vulnerabilities in an AI system before a real adversary does. Microsoft's January 2025 whitepaper defines it as "a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI." Two properties in that definition matter:

• Structured — not ad hoc hacking, but planned around a threat model, a system boundary, and a set of hypotheses.
• Adversarial — the tester deliberately behaves as an attacker or misuser would, not as an ordinary user.

What makes AI red teaming distinct from adjacent activities:

• It is not the same as safety benchmarking. Benchmarks measure a model against a fixed, known evaluation set — useful for tracking progress across models, but bounded by what the benchmark already includes. Red teaming is open-ended: it explores what the benchmark does not cover, including harms and attacks the benchmark authors did not anticipate. Microsoft is explicit that its red team is not a benchmarking function.
• It is not the same as traditional application red teaming. Traditional red teaming targets networks, applications, and identities. AI red teaming targets the same, plus the model itself — its inputs, its outputs, its tool uses, its retrieval sources, and its interaction with humans. Microsoft's own ontology explicitly maps to both MITRE ATT&CK (traditional) and MITRE ATLAS (AI-specific).
• It is not a one-time exercise. Every non-trivial system update, capability addition, tool integration, or retrieval-source change is a candidate for renewed testing. Microsoft's Lesson 8 is direct: "the work of securing AI systems will never be complete."

The output of a good AI red team engagement is not a pass/fail. It is a prioritised set of findings — with reproducible attacks, an assessment of exploitability under the system's actual deployment conditions, and recommended mitigations at architecture, model, and process layers.

The Two Categories of Risk a Red Team Looks For

AI red teams work across two categories of risk, and mature programmes cover both:

• Security risks — the AI system is used to compromise something (data, another system, a user account, a decision). Prompt injection, model extraction, training-data extraction, tool-abuse via agentic frameworks, and jailbreaks that unlock unsafe capabilities all sit here. Microsoft's Lesson 7 — "LLMs amplify existing security risks and introduce new ones" — captures the point: an AI application inherits every traditional security weakness (dependency vulnerabilities, poor error handling, weak authentication) and adds novel ones.
• Responsible-AI harms — the AI system produces outputs that harm users or third parties even without a malicious actor. Bias and discrimination, unsafe advice to vulnerable users, sexualised or violent content, mass generation of misleading content, and defamation of real individuals sit here. Microsoft's Lesson 6 — "Responsible AI harms are pervasive but difficult to measure" — is the frank acknowledgement that this category is harder to quantify but often the more likely one to appear in a real incident.

An AI red team programme that covers only the security side is incomplete. So is one that covers only the responsible-AI side. Enterprise leaders should ask their programme owners which category each exercise is targeting, and why.

The Regulatory and Standards Picture

For years, AI red teaming was largely something responsible vendors chose to do. In 2025 that changed. Three developments now shape enterprise obligations and expectations:

EU AI Act, Article 55. The Act requires providers of general-purpose AI models with systemic risk to conduct adversarial testing, including through red teaming, and to document methodology and results. Systemic-risk designation currently applies to models trained with cumulative compute above 10^25 FLOPs, and can also be conferred by the European AI Office based on other criteria. Timeline: providers placing new systemic-risk GPAI models on the market must comply from 2 August 2025; providers of models already on the market before that date have until 2 August 2027. For enterprises deploying such models, the practical consequence is that adversarial-testing evidence is now something they can — and should — request from their model providers.

NIST AI 100-2 E2025 — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. Published in March 2025, this is NIST's authoritative attack taxonomy for AI/ML systems. It gives red teams and defenders a shared vocabulary — evasion attacks, poisoning attacks, privacy attacks, prompt injection variants — that turns "we tested for prompt injection" into a set of specific, categorised claims. When a red team report cites AI 100-2, both sides know precisely what was tested.

NIST ARIA — Assessing Risks and Impacts of AI. ARIA is NIST's evaluation programme, and its three-tier hierarchy — model testing, adversarial red teaming, and field testing — is a useful template for enterprises structuring internal AI validation. Model testing establishes baseline capability and behaviour under known conditions. Red teaming probes for failure and misuse under adversarial conditions. Field testing observes what happens under actual deployment.

Alongside these, the NIST AI 600-1 Generative AI Profile (July 2024) identifies information security — including prompt injection — as a top-tier risk category for generative AI, and points implicitly to red teaming as the mechanism that discovers such risks.

What Microsoft Learned From 100+ Products

The most substantial public data point on how AI red teaming works in practice at enterprise scale comes from Microsoft. Microsoft's AI Red Team was formed in 2018 — before the current generative AI wave — and by October 2024 had red-teamed more than 100 generative AI products across more than 80 operations. The team's January 2025 whitepaper (Lessons From Red Teaming 100 Generative AI Products, arXiv 2501.07238) distils eight lessons. All are worth reading; five are especially useful for enterprise leaders:

• Lesson 1 — Understand what the system can do and where it is applied. Red teaming without a threat model is theatre. Before adversarial testing begins, the team must know what the system is authorised to do, whose data it touches, and what the highest-impact failure would look like.
• Lesson 2 — You don't have to compute gradients to break an AI system. The most impactful attacks are often the simplest — a well-crafted prompt in an email, a poisoned document in a knowledge base, a social-engineering pattern applied to a chatbot. Enterprises expecting only sophisticated technical attacks under-invest against the ones most likely to succeed.
• Lesson 3 — AI red teaming is not safety benchmarking. Discussed above; worth restating.
• Lesson 4 — Automation can help cover more of the risk landscape. Automated adversarial-prompt generation, fuzzing, and test-harness patterns scale coverage — but do not replace human judgement.
• Lesson 5 — The human element of AI red teaming is crucial. Subject-matter experts (medical, legal, security), cultural competence across regions, and emotional intelligence for evaluating sensitive interactions cannot be automated away.

Microsoft's Takeaway 3 in its accompanying blog post is the strategic recommendation: defence in depth. No single mitigation — no filter, no fine-tune, no system prompt — is sufficient. Red teaming reveals which layers of the defence stack actually catch which failure modes; the mitigations then get distributed across those layers.

What Public-Scale Red Teaming Looked Like — DEF CON 31

The largest public AI red-team exercise to date remains the DEF CON 31 Generative Red Team Challenge, held 11–13 August 2023 in Las Vegas. It gave the field a much-needed data point about scale:

• 2,244 participants tested LLMs from eight vendors — Anthropic, Cohere, Google, Hugging Face, Meta, NVIDIA, OpenAI, and Stability AI — on a platform built by Scale AI.
• Participants exchanged 164,208 messages across 17,469 conversations — evidence that at scale, LLMs produce a very large volume of adversarial signal in a very short time.
• 21 categories of test covered claims to be human, misinformation, bad math, and demographic stereotyping.
• The Wilson Center, Humane Intelligence, and NIST subsequently used the data for research into flaw-reporting practices.

DEF CON 31 was not an enterprise engagement, but it is instructive for enterprise leaders on two points: adversarial testing at scale finds substantial failure signal even with untrained testers, and structured coordination between testers, model providers, and independent researchers produces useable public knowledge — a model enterprises can borrow from for internal programmes.

What Enterprise Leaders Should Ask For

For most large enterprises, the practical question is not "should we do AI red teaming?" — it is "what should we scope, who should do it, and what should the output look like?" A defensible programme addresses at minimum:

• Scope. Which systems (in-house models, third-party APIs, RAG pipelines, agents), which use cases, which categories (security, responsible-AI, or both), which threat models (external attacker, malicious insider, misuse by ordinary user)? Vague scope produces vague findings.
• Depth. How much time and expertise per system? An hour of red teaming per system is a checkbox; two weeks with two specialists per system starts to be credible for anything serious.
• Method. Manual, automated, or hybrid? Any use of NIST AI 100-2 as taxonomy? Any use of ATLAS as adversary-behaviour reference? Any published methodology from a credible programme (Microsoft, NCSC, MITRE, academic) informing the approach?
• Coverage. Which of the twelve NIST AI 600-1 GenAI risk categories are addressed by this engagement, and which are explicitly out of scope? An engagement that quietly omits information-security testing is missing prompt injection.
• Independence. Internal red team, external red team, or blended? External independence matters most when the output must be defensible to auditors, insurers, or a regulator. Internal red teams are cheaper and closer to the system but face conflict-of-interest pressure to under-report.
• Reporting. Reproducible attacks, exploitability assessment against the actual deployment, prioritised recommended mitigations at architecture / model / process layers, and — crucially — a re-test plan for high-severity findings. A red team report that lists findings without exploitability context is unactionable.
• Cadence. Which changes trigger a re-test? Model upgrades, tool additions, new integrations, and material changes to retrieval sources all belong in the trigger list.

Enterprises deploying general-purpose AI models with systemic-risk designation should additionally request, from their model provider, the adversarial-testing documentation required by EU AI Act Article 55 — Annex XI specifies the technical documentation that must be maintained, including descriptions of adversarial-testing methodology and results.

Where Enterprise Programmes Fail

Predictable failure modes appear across engagements:

• Red teaming the model but not the application. Most enterprise AI harms happen at the application layer — the tool scopes granted to the model, the retrieval sources it reads, the actions it can trigger — not at the base model. A red team engagement that stops at the model interface misses most of the real risk.
• Red teaming the golden path. Testing the intended use case in the intended way, and missing the misuse patterns and edge inputs that adversaries actually rely on. This is Microsoft's Lesson 1 in reverse.
• One-time engagements. A single point-in-time report against a system that will be updated every quarter is a decorative artefact. Programmes without cadence are theatre.
• Findings without owners. A finding with no assigned owner, no mitigation plan, and no retest date is a finding that will still be there next year. This is where governance functions must partner with the red team.
• Confusing red teaming with certification. No AI red team certifies a system. It finds failure modes at a point in time. The value is the finding, the mitigation, and the retest — not a green tick on a slide.

What Leaders Should Be Asking

To the CISO: For each production LLM-integrated system, when was the last adversarial test, what did it cover, what did it find, and what has been re-tested since? If the honest answer is "there hasn't been one", the risk position is undocumented.

To the head of AI: Which of our systems currently use general-purpose AI models designated as systemic risk under the EU AI Act, and have we obtained the Article 55 adversarial-testing documentation from those providers? Model-provider evidence is now something enterprises can request, and increasingly are expected to.

To the general counsel and chief risk officer: If an adversarial attack against one of our AI systems caused a material incident tomorrow, could we produce evidence of a documented, credible red-team programme covering that system, or would our position rely on hoping it does not happen? Insurers and courts increasingly want the former.

The Underlying Point

AI red teaming has grown up in three years — from a research activity, to a vendor practice, to an enterprise expectation, to (for systemic-risk GPAI providers) a legal requirement. What has not changed is that no amount of red teaming makes a generative-AI system safe by itself. The point of the practice is not certification. It is the pattern Microsoft names in its final lesson: securing AI systems is never complete, so the organisations that fare best treat red teaming as the mechanism that keeps their defence stack honest — surfacing what is currently broken, in what order to fix it, and what to test next.

For enterprise leaders, the useful question is not whether AI red teaming is real work. The regulator, the insurer, and the customer have already answered that. The question is whether your programme finds the things a determined outsider would find first — and whether the findings change what your organisation does next.