Take a hypothetical. You look after a client your firm has tested for six years. This morning, its head of security emails you: “We’re launching an AI assistant next month. Can you red team it?”
You know the client’s web application, its application programming interfaces (APIs), and its cloud setup. Your firm hasn’t sold AI work before. You could treat the request as a familiar job with a new label and send a quote the same afternoon.
Hold the quote until after one call. “Red team it” could mean a security check before launch, a review of what the assistant says to customers, evidence for an EU AI Act file, or a full attack simulation against the client’s defences. Each one needs a different scope, a different team, and a different report. So sort the request before anyone scopes it: what your existing pentest covers, what needs specialist AI scoping, and what sits outside the engagement.
The Client’s Wording Won’t Tell You the Scope
The term has stretched. Microsoft’s guide to planning red teaming for large language models says red teaming “has extended beyond traditional cybersecurity and evolved in common usage to describe many kinds of probing, testing, and attacking of AI systems.” The same guide lists hate speech, violence, and sexual content as harms to test for, which puts content safety inside the term.
Vendors draw the line in different places. 7ASecurity describes AI red teaming as simulating “a targeted attack to test your detection and response capabilities”, with a scope that reaches the security operations centre. Equixly gives AI red teaming a narrower target, “the model’s behavior”, and leaves the system around the model to AI penetration testing.
Your client picked a reasonable word. It still can’t tell you what to test. Treat “AI red teaming”, “AI pentest”, and “AI Act testing” as the start of a conversation. If you want definitions of each discipline, read What Is AI Red Teaming and Adversarial Testing.
Three Questions for the First Call
You don’t need an AI threat model for the first call. You need three answers, written down in the client’s words.
1. Who uses the system, what can it read, and what can it do? Ask who has access and whose identity the system acts under: each user’s own, or a shared service account. Then ask for the data sources it retrieves from, such as documents, mailboxes, or a customer database, and the actions it can take, such as sending email or updating records. An assistant that answers from a public FAQ and an agent that can issue refunds need different scopes.
2. Which decision does the client face? It might be a launch decision, an answer to a customer’s security questionnaire, a board update, or an AI Act evidence file. The decision sets the depth of the work and the deadline.
3. Who will read the evidence? Engineers want findings they can reproduce and fix. A governance lead or auditor wants to see what was tested, against which criteria, and with what result. A customer reviewing a questionnaire wants a short answer it can rely on.
These answers sort the request into parts. They don’t scope the AI work, and you shouldn’t try to make them do that.
Six Example Requests and Where Each Part Goes
Here are six requests a pentest firm is likely to hear, with the first question to ask and a starting split of the work.
| The client says | Ask first | Your pentest | Plan with an AI specialist | Outside this engagement |
|---|---|---|---|---|
| “Can you red team our AI before launch?” | Who uses it, what can it read and do, and as whom? | Authentication, sessions, API access checks, infrastructure | Instructions hidden in input or retrieved content, tool misuse | Harmful-content testing, unless agreed and staffed separately |
| “We need AI Act testing” | Who classified the system and your role? Who reads the evidence? | Security testing already in the evidence file | Adversarial testing for the articles that apply | Legal classification, conformity assessment |
| “Is our Copilot rollout safe?” | Which data sources, connectors, and agents are on, and who set the permissions? | Tenant identity, access, and sharing settings | Exposure through Copilot’s retrieval and connectors, instructions in shared content | Research into Microsoft’s own service |
| “Can you jailbreak-test the model?” | Are you choosing a model, or securing an application built on one? | The application around the model | The application’s AI controls under attack | Model selection benchmarks |
| “Check our AI for bias” | What outcome do you suspect, in normal use or under deliberate manipulation? | Usually none | Attackers manipulating outputs | Fairness evaluation |
| “A customer sent us an AI security questionnaire” | Which questions can’t you answer with existing documents? | Questions your reports already answer | Questions that need tested evidence | Certification or compliance claims |
Read the “Your pentest” column first. In five of the six rows, part of the work is testing your team already does, and you keep it in your scope.
Treat the columns as a starting split for a firm without a mature AI practice. Agree the final ownership against the controls you find. Take one user reaching another user’s data. A missing access check on the API behind the assistant is familiar pentest work. A planted document that steers the assistant into fetching another user’s record needs AI expertise. An attack path that uses both needs both teams, reviewing it together.
The last column marks the limits of the engagement you’re proposing and of your team’s skills. Some security firms have fairness or responsible-AI specialists, and if yours does, bias work can sit inside your offer. Testing whether an attacker can manipulate outputs doesn’t replace a fairness evaluation. For hosted services such as Copilot, you can test the client’s own tenant and integrations with its authorisation. Microsoft’s Security Testing Rules of Engagement apply to the owner of the resource and to consultancies acting for it, so check them before you start. Research into the provider’s service is a separate activity, and any flaw you find in it goes to the provider through coordinated disclosure.
The AI Act row needs the most care. The Act names adversarial testing for one group: providers of general-purpose AI models with systemic risk, whose model evaluation must include “conducting and documenting adversarial testing of the model” under Article 55(1)(a). For high-risk systems, Article 9 requires testing as part of the provider’s risk management, and Article 15 covers accuracy, robustness, and cybersecurity. For systems classified as high-risk under Article 6(2) and Annex III, these duties apply from 2 December 2027 under Regulation (EU) 2026/1744.
Which articles matter depends on classification and on the client’s role. Articles 9 and 15 may apply if the client is the provider. A deployer has its own duties under Article 26, including use in line with the provider’s instructions, human oversight, monitoring, and keeping logs, so the evidence it needs can differ. Legal counsel can advise on classification, and the client’s legal or governance owner confirms it, using technical facts about the system. Ask who has done that before you agree to work with the Act’s name on it. For the wider picture, read What the EU AI Act Requires From High-Risk AI Systems.
Where Specialist Scoping Comes In
Your team can test a large part of an AI system. An API that feeds an assistant can have the same authorisation flaws as any other API, and your testers know how to find them. The AI part of the scope depends on six things an endpoint list doesn’t show you. Each one needs a decision before testing starts, and each one changes what the client can do with the report.
| The scope has to settle | Why it matters |
|---|---|
| The instruction paths in scope | An attacker doesn’t need the chat box. A document in the search index, an inbound email, or a support ticket can carry instructions the assistant follows. In June 2025, Aim Security disclosed EchoLeak (CVE-2025-32711), where a single crafted email could lead Microsoft 365 Copilot to send internal data to an attacker-controlled server. A scope that only covers typed prompts misses that path. |
| Who can see what when the assistant retrieves data | The assistant might fetch documents with each user’s own permissions, or with broader permissions than the requesting user has. A controlled test uses test users with different access levels and planted records, so the team can show what one user can reach. The scope also separates two causes: permissions already set too wide in the source system, and the assistant retrieving or revealing data it shouldn’t. |
| What each tool can do, and under which identity | An agent that sends email or changes records under a service account can cause more harm than one that can only read. The scope decides which actions run live, which run in a test environment, and which stay switched off. |
| How many attempts each test case gets | The same input can produce different outputs. One failed attempt doesn’t show the system is safe. The scope sets the number of attempts per test case. The report gives reproductions out of attempts, for example 3 of 20, with the tested configuration and how the system was reset between attempts. Don’t present that figure as an attacker’s chance of success. |
| Which provider rules apply | Hosted models and services such as Copilot come with the provider’s testing rules. The scope separates what you test in the client’s environment, with its authorisation, from what you report to the provider. |
| The success criteria, agreed before testing | If the results will support an AI Act file for a high-risk system, Article 9(8) calls for testing against “prior defined metrics and probabilistic thresholds”. Agree those criteria with the client before testing starts. Meeting them supports the file. It doesn’t satisfy the article on its own. |
Some of these answers won’t exist yet. If nobody can describe the configuration or the evidence the client needs, don’t promise a final scope or price. Propose a short discovery step first.
How the Joint Scoping Works
You stay the client’s security provider throughout. You bring the client relationship, the answers from your first call, and your conventional scope. The specialist brings the AI test cases, the attempt counts, the provider-rule check, and the evidence format. Together you work towards one scope the client signs, with a named lead for each test case. That can take more than one session if the first one turns up missing information.
Before the report goes out, both teams reconcile what was covered and review any attack path that crosses both parts of the scope. Independent weaknesses stay separate findings, cross-referenced, with the combined path documented. For an example of a combined scope on paper, read Scoping an AI Assessment Alongside a Pentest.
Provion offers this scoping support to pentest firms. Bring your first-call notes and your existing scope, and we’ll define the AI part of the scope with you, while the client relationship stays with you. To compare the ways firms deliver the AI work once it’s scoped, read Build or Partner.
Bring the Request to a Partner Conversation
Provion is an adversarial assessment platform for AI systems, built for security teams in Europe operating in security-conscious and regulated environments. We offer pentest firms scoping support for the AI part of a client request, and the firm keeps the client relationship and its own test.
If a client has asked you for AI red teaming or AI Act testing, book a partner call and bring the request as they wrote it. We’ll scope the AI workflow with you, alongside the test you already run.
