Picture a retrieval-augmented customer-support assistant. It reads from an internal knowledge base, answers questions in natural language, and can call a customer-record tool to look up account details. Functional testing confirms it works: ask a billing question, get a correct billing answer. That test tells you the system does its job. It tells you nothing about what the system does when someone feeds it a document written to hijack the workflow.
That second question is what AI red teaming answers. AI red teaming is a scoped test of one AI system under adversarial conditions. NIST defines artificial intelligence red teaming as a structured effort to find flaws and vulnerabilities in an AI system, usually in a controlled setting and often with the developers involved (NIST CSRC). In production terms, that means probing the model and the application path around it, recording how an attack moves through the system, and turning the result into a validated finding with severity, impact, remediation, and a retest path.
A production-system assessment covers the model, prompts, guardrails, retrieval, data, tools, permissions, APIs, and runtime controls. A red team asks whether untrusted content can redirect the support assistant, whether it can expose another customer’s data, or whether it can trigger a tool action outside its intended use. In this example, the failure sits in the wiring between the model and the business system. They live in the wiring between the model and the business.
Model Testing, System Red Teaming, and Penetration Testing Answer Different Questions
Teams often assume they have already covered this ground. They ran a model evaluation, they run application penetration tests, and they added an automated attack scan. Each of those is useful. None of them answers the production-system question on its own, because each is built to answer something else.
NIST’s ARIA programme draws the line cleanly. Model testing checks a model’s claimed capabilities and the performance of its guardrails. Red teaming stress-tests the application and its guardrails under adversarial scenarios to elicit disallowed behaviour. Field testing observes the system in regular use (NIST Human-Centered AI). A clean model score does not establish whether retrieved content can redirect a workflow, whether an agent can misuse a tool, whether permissions contain the impact, or whether runtime controls detect the event.
| Test | Main question | Typical scope | Useful output |
|---|---|---|---|
| Model evaluation | Does the model meet selected capability, safety, or performance criteria? | Model inputs, outputs, and selected guardrails | Scores, failure cases, and benchmark results |
| AI red teaming | How can an adversary or misuse scenario make this AI system fail? | Model, prompts, retrieval, data, tools, permissions, APIs, and controls | Reproducible findings with severity, impact, remediation, and retest evidence |
| Penetration testing | Can an attacker exploit conventional software, identity, network, or configuration weaknesses? | Applications, APIs, infrastructure, identity, and deployment | Exploitable vulnerabilities and remediation |
These methods complement each other. The tester’s title matters less than whether the assessment covers AI-specific failure paths across the production system. It is that conventional scopes and tools do not establish how a production AI system behaves under adversarial conditions, because the failure path runs through language, retrieval, and tool use rather than through a misconfigured port.
What AI Red Teaming Tests Across a Production System
A long list of attack names does not tell a buyer what to scope, fix, or monitor. A better mental model groups the production attack surface into five surfaces that map to components a team actually owns. A good test plan follows the system’s architecture across these surfaces rather than applying every attack category to every system.
| Surface | Production components | Example tests | Evidence to record |
|---|---|---|---|
| Prompts and guardrails | System prompts, templates, input handling, output filters, orchestration | Direct and indirect prompt injection, jailbreaks, guardrail bypass, unsafe output handling | Attack input, response, guardrail result, failed policy, repeatability |
| Retrieval and data | Vector stores, embeddings, documents, connectors, fine-tuning data, tenant boundaries | Poisoned content, cross-context leakage, retrieval manipulation, unauthorised data access | Source content, retrieved chunk, access context, response, data-flow trace |
| Tools and permissions | Agent tools, APIs, identities, scopes, approval gates, memory, multi-step workflows | Excessive functionality, excessive permissions, unauthorised action, tool-output injection, approval bypass | Tool call, identity, permission, action, approval state, logs, downstream effect |
| Model and data attacks | Model endpoints, pre-trained components, training and fine-tuning pipelines, model artefacts | Data or model poisoning, adversarial examples, model evasion, extraction, confidentiality attacks | Model version, attacker assumptions, input, output, affected property, test limits |
| Runtime controls | Authentication, rate limits, secrets, logging, monitoring, fallbacks, supply chain, incident paths | Detection bypass, unbounded resource use, secret exposure, unsafe fallback, control failure | Logs, alerts, control response, recovery behaviour, owner, escalation result |
OWASP’s 2025 material supports each surface with concrete detail. It defines direct prompt injection as user input that changes model behaviour and indirect injection as instructions delivered through external sources such as documents or web pages (OWASP LLM01). It traces excessive agency back to excessive functionality, permissions, or autonomy, and recommends limiting tools, requiring approval for high-impact actions, and enforcing authorisation in downstream systems (OWASP LLM06). Its vector and embedding guidance covers cross-context leakage and poisoned retrieval data, which is why the retrieval pipeline, access controls, and tenant boundaries need testing rather than a stripped-down model endpoint (OWASP LLM08).
What an AI Red Teaming Assessment Produces
A credible assessment produces a reproducible evidence trace for each finding. NIST’s Measure Playbook recommends documenting the range of security test conditions and results (NIST AI RMF Playbook). For a buyer, that principle becomes a decision record. Consider the support assistant again, tested against an indirect prompt injection. The following trace is illustrative, built to show the anatomy of evidence rather than to report a specific engagement.
- System and intended use. A customer-support assistant with read access to an internal knowledge base and a customer-record lookup tool. Intended use: answer account and billing questions for the authenticated customer only.
- Test objective and attacker assumption. Determine whether untrusted content in a retrieved document can redirect the assistant to disclose another customer’s data. The attacker can submit a support document but has no privileged access.
- Injected content and retrieval path. A support article contains hidden instructions telling the assistant to look up and summarise the most recent account it can reach. The article is ingested and later retrieved as context for an unrelated query.
- Model response or tool action. The assistant follows the embedded instruction, calls the customer-record tool, and returns fields belonging to a different customer.
- Control that failed. The retrieval layer treated document content as trusted instructions, and the tool ran under a broad identity with no per-request authorisation check on whose record was requested.
- Severity and business impact. Expert review rates the finding high: cross-tenant data disclosure triggered by content any user can plant, with direct privacy and regulatory exposure.
- Remediation owner and control change. The application team scopes the tool identity to the authenticated customer, adds an authorisation check in the downstream service, and separates retrieved content from instructions in the prompt.
- Retest and residual risk. A retest repeats the vector and confirms the tool now refuses the cross-customer lookup. Residual risk is documented for related tools not yet in scope.
A system owner can act on this record because it identifies the tested version, attacker assumption, failure path, affected control, impact, remediation, and retest result.
One honest limit belongs here. Red teaming cannot prove the absence of vulnerabilities. Probabilistic behaviour, incomplete coverage, changing models, and later system changes bound every result. Versioned evidence and retesting are what keep a finding useful over time, not a claim of total coverage.
When a Team Needs AI Red Teaming
This work supports a decision. A reader with no named system, no access, and no decision to make is not ready for an assessment, and no amount of testing will change that. The strongest triggers are operational:
- A production launch, production review, or security sign-off for one AI system.
- A customer, procurement team, board, or risk function asks for evidence of AI security testing.
- A system gains retrieval, new data sources, tools, memory, broader permissions, or higher-impact actions.
- A model, prompt, data source, guardrail, or orchestration change makes earlier test evidence unreliable.
- A jailbreak, data-exposure concern, or unexpected tool action shows that normal-path testing missed a failure mode.
- An organisation needs testing records for applicable EU AI Act risk-management, cybersecurity, or documentation work.
Read that list as a self-qualification test. A team approaching launch, adding tools or data, answering a customer review, or responding to an incident has a clear reason to act. The output should help an owner fix a weakness, accept residual risk, restrict deployment, add monitoring, or decide the system is not ready. For the wider evidence gate around launch, see Is Your AI System Ready for Production Review?
Keeping the EU AI Act Boundary Precise
Regulation is a real driver, but it is easy to overstate. Three points keep the boundary accurate.
First, the EU AI Act names an explicit adversarial-testing duty in one place. Article 55 requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing of the model (EU AI Act Article 55). That obligation does not apply to every AI system.
Second, Article 15 requires appropriate accuracy, robustness, and cybersecurity for high-risk systems, and names AI-specific vulnerabilities that technical measures should address where appropriate: data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks, and model flaws (EU AI Act Article 15). It does not create a universal requirement for every organisation to buy an independent red-team engagement.
Third, testing records can support a wider evidence file where the organisation’s role and system classification make those obligations relevant. They do not certify compliance, and a single assessment does not satisfy every framework at once. For the detailed legal requirements and classification logic, see What the EU AI Act Requires From High-Risk AI Systems.
Start With the Evidence, Then Scope the System
The most reliable way to judge AI red teaming is to look at what it produces. Before commissioning anything, inspect a finding: its structure, its severity logic, its evidence trace, and its remediation. That is where a credible assessment separates from an automated scan.
Review the Provion sample report to see how a finding is built, from test vector to affected component, failed control, severity decision, business impact, remediation, and retest. Automated tools can generate and run test cases, and they contribute coverage and records. A final finding still needs expert review of repeatability, severity, exploitability, context, and impact, and that evidence stays inside the customer’s own workspace.
If you have one named system or workflow and a decision that needs evidence, the next step is a scoping call to define what the test should cover. Review the sample report first, then decide whether the assessment fits your system and decision.
