Your team has tested a client’s web application and API for three years. Now the client has added a customer-support AI assistant. It answers logged-in customers using the support documentation and a tool that looks up their account record.
The client’s security lead asks one more question: could the assistant show one customer another customer’s data?
Answering it starts with scope. Before anyone writes a test case, agree what your pentest already covers, what’s new, where testing stops, who does what, and what evidence comes back. This article assumes you’ve already decided to offer AI assessments. If you haven’t, start with our build or partner decision framework.
Name the Client Decision and the AI Workflow
Start with the decision the answer will feed, such as releasing the assistant to all customers or keeping the account lookup tool switched on. Write it down, with the name of the person at the client who owns that decision.
Then pick the right kind of work:
- If your current engagement can answer the question, add test cases to it.
- If the answer depends on how the model behaves across document retrieval, tool calls, and multi-turn conversations, the client needs a separate AI assessment scope.
- If the client can’t describe the configuration that will ship, propose a discovery phase first.
If the client already knows about a defect, such as the lookup tool skipping a per-customer check, fix that first. Testing can still help in the meantime: authorised testers can validate the defect’s impact, or test the paths it doesn’t affect.
Map the Existing Scope Against the AI Questions
Put your existing pentest scope next to the AI questions for the same system. Authentication, sessions, and the account API’s authorisation checks are familiar ground, and the OWASP Web Security Testing Guide (stable release 4.2) gives your team a shared structure for them. Familiar ground isn’t proof of coverage, though. Check the scope document and the last report before you mark anything as covered.
The new risk sits in the path between components. In August 2024, PromptArmor demonstrated an indirect prompt injection against Slack AI: instructions planted in content the assistant later reads. An attacker in the same workspace posted instructions in a public channel. When a victim asked Slack AI about a private channel the attacker couldn’t read, the answer carried that private data inside an attacker-crafted link, and one click sent it to the attacker’s server. No channel permission was bypassed. The assistant put data the victim could see into output the attacker controlled. Salesforce said it had deployed a patch and had no evidence of unauthorised access to customer data.
| Area | Existing pentest question | Additional AI assessment question | Evidence to agree |
|---|---|---|---|
| Application and API | Can a user bypass authentication or reach another account? | Can the assistant be pushed into requesting data outside the user’s entitlement? | Test identity, requested record, the API’s authorisation decision, returned content |
| Guardrails and policy | Do input checks and filters block malicious or out-of-policy input? | Can a reworded, multi-turn, or role-play request get the assistant past its refusal policy and towards the protected record? | Prompt sequence, policy decision at each turn, final response, whether the bypass repeats |
| Retrieval | Who can add or read documents, under the application’s access controls? | Can instructions inside a retrieved document redirect the assistant into disclosing data it shouldn’t? | Document source, retrieved content, assistant response, relevant control decision |
| Connected tool | Does the tool API enforce the caller’s permissions? | Can the assistant call a permitted tool with parameters that break the intended workflow? | Tool arguments, identity used, API result, observed effect |
| Output and evidence | Does the application handle output and log relevant security events? | Does the final answer expose protected content, and can both teams trace how? | Recorded response, synthetic canary references, event correlation, known visibility gaps |
Treat the rows as scoping questions, not a test plan. Mark each one covered, partial, or open, and cite the scope document or report that shows it. Keep them tied to one system: a permissions flaw and a model behaviour can feed the same attack path, so don’t split them into two engagements by default.
A guardrail is one control among several. If testers get the assistant past its refusal policy, the policy has failed. The account API must still enforce customer entitlement whatever the model asks for. OWASP’s Excessive Agency guidance says the same: implement authorisation in downstream systems instead of relying on the model. Keep three outcomes apart in the scope and the report:
- The assistant attempts a tool call for another customer’s record.
- The API denies it.
- Protected content reaches the wrong customer.
Only the third is a confirmed disclosure.
Define the Test Boundary and Required Access
Name the exact configuration you’re testing. The assistant’s behaviour depends on its model, system instructions, document collections, identity handling, and tool permissions, and so does your estimate. Describe the proposed configuration during scoping. At kickoff, record the baseline your testers will test, and agree how a material change mid-test affects effort and what the report can conclude.
Record the access your testers need:
- Identities: at least two synthetic customers, so testers can attempt cross-customer access between known accounts.
- Data sources: the documentation collections in scope, and whether testers may add attacker-written documents to a test copy of the index.
- Tools: the account lookup tool and its permitted actions. Exclude anything that writes or sends unless the client authorises it.
- Environment: a staging deployment. Name every difference from production (identity, data separation, document sources, tool permissions, outbound access, guardrails, logging) and the production conclusions each difference limits.
- Data separation: synthetic records only, with test identities that can’t reach production data.
- Exclusions: third-party model training infrastructure, production data, and denial-of-service testing unless agreed.
As in a conventional test, the client signs written authorisation for the agreed scope. Set stop conditions in advance. Testers pause and escalate if they retrieve a real customer record, reach a system outside the boundary, or trigger an external effect the client hasn’t approved. A callback to a synthetic canary endpoint can be part of a controlled test, so list approved external effects in the authorisation. Agree where testers store prompts, responses, and tool logs, and when they delete them.
Our guide on how to prepare for an external AI red teaming assessment covers the client’s preparation pack.
Assign Testing and Delivery Responsibilities
Two teams testing one system need a written split before kickoff. Treat this table as a starting point; some lines will change by engagement.
| Responsibility | Proposed arrangement to agree |
|---|---|
| Client relationship and wider engagement | Pentest partner owns the commercial relationship and client communication |
| AI scope and test plan | Partner and specialist agree coverage, overlap, access, and exclusions |
| Authorisation and environment | Client authorises the work and provides the environment; testers stay inside that boundary |
| Specialist execution and validation | Named assessors do the agreed work and record their conclusions |
| Combined reporting and readout | Partner and specialist reconcile findings and say who contributed what |
| Remediation and risk acceptance | Client assigns fixes and accepts residual risk; assessors give guidance and agreed retesting |
Name who leads each technical question. For example, your team leads the account API authorisation tests, the specialist leads document injection and multi-turn guardrail testing, and both work the combined path. Also name who validates a finding before it reaches the client, and who can challenge a severity rating.
Keep one route to the client. Agree who talks to the system owner about schedule and scope changes, and who joins a technical readout. Decide in writing who calls the client, and within what time, if testers confirm a cross-customer disclosure.
Keep attribution clear. If a specialist delivers part of the AI scope, the report should say who performed and who reviewed each part, including under your branding. The client needs to know which method produced a finding and who stands behind it.
Agree the Evidence and Report Handoff
Agree the finding structure before testing starts, so both teams’ results fit one report. For each finding, record:
- A finding reference shared across both teams
- The system baseline: configuration, model version, index state at test time, and any resets between attempts
- The observed behaviour: prompt sequence, retrieved content, tool calls, control decisions, and response
- The affected control, such as the account API’s authorisation check or the assistant’s refusal policy
- The outcome: attempted action, denied access, or confirmed disclosure
- The severity, its basis, and who set it
- The impact in this deployment, and who validated the finding and how
- A remediation recommendation, the issue owner, and the retest criteria
Model output varies between runs. A confirmed disclosure shows the system failed under the recorded conditions. Before you count a synthetic canary value in a response as unauthorised access, rule out exposure in an earlier prompt, contamination from a shared session, and fabricated output. Report successful reproductions against total attempts, with the baseline and any resets. That fraction describes this test. It doesn’t estimate how often a real attacker would succeed. If the logs miss a step in the path, say so in the report.
Cross-scope findings need a coordinator. Suppose your team finds that the account API accepts another customer’s identifier without checking entitlement, and the specialist shows that instructions in a retrieved document can steer the assistant into supplying one. That’s two weaknesses, an authorisation flaw and a trust-boundary flaw, with separate fixes, owners, impact, and retest criteria. Report them as separate, cross-referenced findings and document the combined attack path. Merge findings only where both teams describe the same underlying issue, and agree in advance who coordinates the chain.
Automated tools can produce many candidate results. Label them as automated outcomes until an assessor has reviewed them, and report only validated findings as findings. Our breakdown of seven gates to a validated finding sets out one approach, and our overview of what an external AI red teaming assessment includes shows the buyer’s side of the handoff.
Where EU AI Act Article 15 Fits
Clients in EU-regulated sectors may ask how this work relates to the EU AI Act. Article 15 requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity. Article 15(5) requires them to be resilient against unauthorised attempts to alter their use, outputs, or performance by exploiting vulnerabilities. It names data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks, and model flaws as AI-specific vulnerabilities for technical measures to address, where appropriate to the circumstances and risks.
Under Regulation (EU) 2026/1744, those obligations apply from 2 December 2027 for systems classified as high-risk under Article 6(2) and Annex III, and from 2 August 2028 for those under Article 6(1) and Annex I (timeline and evidence to prepare).
For a high-risk system, validated findings can contribute to the evidence behind the client’s Article 15 cybersecurity documentation. They don’t cover accuracy or the wider conformity process, and the assessment doesn’t establish compliance. The example assistant isn’t presumed high-risk. Classification follows the criteria in Article 6 and the system’s intended purpose, and the client’s legal or governance function confirms whether the obligations apply and in which role. Our knowledge base sets out both classification routes.
Write the Scope Addition Before Pricing the Work
Put the result on one page and agree it with the client before you commit to a price. Cost and duration depend on the configuration, the environment, and the access the client can provide. A provisional estimate can go out earlier if it lists its assumptions; revise it if a material assumption changes.
| Field | Support assistant example |
|---|---|
| Client decision and system owner | Whether to release the assistant to all customers; owner named by the client |
| Workflow and configuration | Customer-support assistant. Model, system instructions, documentation collections, identity handling for the lookup tool, and read-only tool permissions as described at scoping; tested baseline recorded at kickoff |
| Question | Can the assistant show one customer another customer’s information? |
| Coverage status | Authentication and account API authorisation: covered, per the existing scope document. Guardrail bypass: partial. Retrieval injection and tool parameters: open |
| Hypothesis 1, direct | A logged-in customer uses reworded, multi-turn, or role-play requests to make the assistant request another customer’s record. Expected control: the account API denies the call on entitlement, whatever the model requests |
| Hypothesis 2, indirect | An attacker plants instructions in an indexed source, such as a community forum the client ingests. When another customer’s question retrieves them, the assistant puts that customer’s record into output the attacker controls, such as an external link. Expected controls: output handling blocks unapproved external links, and the lookup tool returns only the session customer’s record |
| Excluded work | Third-party model training infrastructure; production data; denial-of-service testing |
| Environment, identities, and permitted actions | Staging with synthetic records only and named differences from production; two or more synthetic customers; read-only lookup tool; approved canary endpoint |
| Execution limits | Selected workflows and roles; repetitions per test case or an agreed test budget; rate and concurrency limits; discovery allowance; retest allowance |
| Responsible testers and escalation authority | Named per engagement |
| Evidence and report handoff | Shared finding structure; separate, cross-referenced findings with the combined attack path; combined report and joint readout |
| Prerequisites, timing, and retest conditions | Agreed per engagement |
Rows both parties sign become contracted work. Questions left only in the matrix stay out of scope until someone adds them. List anything unresolved, such as a staging environment without production guardrails, and schedule discovery before testing until it’s closed.
Bring One Client Workflow to a Scoping Conversation
Provion is an independent adversarial testing service for AI systems, built for teams in security-conscious and EU-regulated environments. It produces structured, defensible evidence of how an AI system behaves, fails, and exposes risk. We work with pentest and offensive-security firms that want to add AI adversarial assessment alongside their existing engagements. We start with scoping support for one named client system, and agree the coverage, evidence, and responsibilities for that engagement before any work begins.
Bring one client AI workflow and your existing pentest scope. We’ll discuss the additional assessment questions, evidence, and delivery responsibilities with you.
