Your AI inventory has 30 rows, and your budget covers one external assessment this half. Five of those rows compete for the slot: an HR policy assistant, an accounts-payable agent that reads supplier invoices, a coding agent with continuous integration (CI) access, a loan pre-screening assistant, and the AI drafting feature your vendor switched on inside the CRM.
Use the worksheet below to choose where an assessment would resolve a consequential evidence gap. It covers large language model (LLM) applications, copilots, agents, and enabled vendor AI features. Depending on the gap, the next action may be internal validation, vendor evidence review, or remediation.
Rank AI Systems by Exposure and Evidence
Project visibility and model sophistication don’t establish assessment priority. A launch date helps you schedule work but doesn’t measure exposure. A board request carries weight when it names a decision and the evidence that decision needs, such as whether to extend an agent to a second business unit.
Keep statutory classification separate from security exposure. A system outside the EU AI Act’s high-risk categories can still let an attacker trigger a consequential, unauthorised action. Classification and your organisation’s role as provider or deployer shape your assurance obligations. Neither gives you a security ranking.
Gather the Minimum AI Inventory
The NIST AI Risk Management Framework (AI RMF) Playbook recommends, under GOVERN 1.6, mechanisms to inventory AI systems, resourced according to organisational risk priorities. GOVERN 6.1 covers policies for third-party AI risk. Both are voluntary guidance.
Use one row per named system or workflow, and record:
- Owner, intended use, deployment status, and user groups
- Model or provider, data sources, tools, effective permissions, and the identity the system executes under
- Critical controls, the evidence behind them, the configuration and scope that testing covered, and material changes since testing
- Known findings, incidents or near-misses, and open unknowns
- The decision the evidence must support, its owner and date, or the urgent operating decision you need now
Include the vendor AI features your teams have enabled. Start with the inventory you have and label the gaps, because serious exposure won’t wait for a perfect catalogue. The System Definition Evidence checklist in our production review readiness guide covers the system boundary in more depth.
Ask Three Questions About Each AI System
Exposure
Trace the paths untrusted content can take into the system: user prompts, emails, uploaded files, web pages an agent browses, retrieved documents, and tool outputs. Separate direct prompt injection, where a user types the instruction, from indirect injection, where the instruction arrives inside content the system reads. The OWASP Top 10 for LLM Applications 2026 covers both under LLM01 Prompt Injection. Internal sources can carry attacker-authored material too, such as uploads to a shared drive.
Consequence
Next, ask what that influence could cause: data disclosure, unauthorised actions, corrupted outputs or records, service disruption, and resource or cost exhaustion. Follow the output to the people and systems that act on it downstream. A read-only assistant for an internal audience can still feed a decision someone makes about a payment or a loan.
Assess the system’s effective authority, the users or tenants it can reach, whether you can reverse the effect, and what it costs the business. OWASP LLM02 Sensitive Information Disclosure and LLM03 Excessive Agency cover two of these consequence classes. For tool-connected agents, compare the permissions the design declares with the authority the assembled workflow can exercise. Our MCP security testing article traces that gap from input to business effect.
Evidence
Then ask what supports the controls between the attacker and the consequence. Check whether the evidence covers the current configuration, identities, permissions, and consequential attack paths. A recent report date means little if the system gained a tool after testing.
Compare two systems with similar exposure. One has test results that validated its approval boundary against the current configuration. The other has a generic vendor report about the base platform. Another assessment may resolve little for the first and the main question for the second. Missing evidence raises your uncertainty. It doesn’t prove anyone has exploited the system.
Use the Rule of Two as an Escalation Signal
Meta’s Agents Rule of Two describes three properties an agent shouldn’t combine in one session without supervision: processing untrusted input, accessing sensitive systems or private data, and changing state or communicating externally. We use that combination as a signal to examine the attack path and its controls in detail, without granting the system automatic first place. Meta pairs the rule with supervision or reliable validation where a workflow needs all three, and limits what it claims to protect against.
Simon Willison points out that untrusted input plus state-changing capability can cause harm without any private-data access. A system with fewer than all three properties hasn’t proven itself safe either.
Use a Triage Worksheet With Explicit Decision Rules
Use this worksheet to structure a prioritisation discussion. It is a qualitative decision aid, not a validated risk model. NIST AI RMF MANAGE 1.2 supports prioritising risk treatment by impact, likelihood, and available resources or methods.
| Worksheet field | What to record |
|---|---|
| Exposure | Who can introduce or modify content, through which channels, and which privileged context, data, or actions that content can reach. |
| Understanding of the path | Known, partly understood, or unknown. Name the unverified steps and what would establish them. |
| Plausible consequence | Limited and recoverable. Material within a bounded process. Severe or hard to reverse across sensitive data, systems, or decisions. |
| Control evidence | Applies to the current critical path. Partial, or the system changed since testing. Missing or unknown. |
| Timing | Immediate operating decision. Named decision with a date. Planned reassessment with an owner and a review trigger. |
Record a concrete scenario, such as “externally authored invoices reach the agent’s context”. A well-documented path can still carry serious exposure. Keep your understanding of the path separate from its reach, and consider user population, autonomy, change rate, and regulatory role.
Apply the rules in order:
- Escalate suspected compromise, established serious exposure, or uncertainty that prevents a defensible decision to keep operating. Use existing incident-response or risk-acceptance authority.
- Identify consequential attack paths with weak or missing control evidence. Investigate unknowns before you treat a system as low priority.
- Choose the work that resolves the uncertainty: containment, remediation, targeted testing, a broader assessment, or vendor evidence review.
- Schedule against operational urgency, the decision date, and the expertise available. Record any deferral’s reason and review trigger.
If two systems both warrant assessment, compare the consequences of delaying each one, the evidence needed to support its next decision, and whether the assessment can produce that evidence in time. Give priority to the system where delay leaves the more consequential decision unsupported. Record the reason for deferring the other system and any temporary restrictions.
Choose the Action and the Assessment Route
A high-risk system that isn’t ready to test stays high priority. Manage the immediate exposure while its owner resolves the access, environment, or evidence gaps needed for testing. Then decide who should perform the assessment.
Contain or Remediate an Established Weakness
Address an established consequential exposure before commissioning a broad assessment merely to rediscover it. Restrict or suspend the capability where warranted, preserve evidence, name an owner, and set a validation date. If the weakness or its reach is uncertain, targeted investigation may come first. Suspected compromise goes to incident response.
Typical changes narrow credentials, restrict tools or outbound destinations, and enforce approval for high-impact actions. Approval needs an informed approver and enforcement outside the agent’s discretion. Session separation has to stop untrusted instructions from reaching privileged context through messages, memory, or other hand-offs. You can’t treat a listed control as evidence that it works, so record the new configuration and validate the relevant path after the change.
The multinational guidance Careful Adoption of Agentic AI Services supports least privilege, distinct agent identities, and controlled deployment. It also recommends restricting agentic AI to low-risk, non-sensitive tasks, so read its phased-deployment advice with that limit attached.
Commission an Independent Assessment
Justify independence by an internal expertise or capacity gap, a need to challenge internal assumptions, or a customer or governance requirement for independent evidence. A high-risk label alone doesn’t settle the delivery route.
Before testing starts, define the uncertainty to resolve, the decision it supports, the access testers need, and a representative environment. Capture the system state, freeze the material changes you control during testing, and name the provider-side changes you can’t freeze, such as a hosted model update.
In the arXiv preprint The Attacker Moves Second, Nasr et al. report bypassing 12 evaluated defences against jailbreaks and prompt injection, with attack success above 90% for most of them. The study supports evaluating defences against adaptive attackers. It doesn’t establish a general failure rate or remove the value of static regression tests.
Use Internal Validation or Vendor Evidence Review
Match the work to competent staff and to the evidence gap you have. A vendor SaaS feature may require documentation requests, configuration checks, and authorised access before anyone can test it directly. Our breakdown of a validated AI red team finding sets out the validation discipline internal teams can apply.
Monitoring counts as a decision when it has a documented rationale, an owner, observable signals, and an escalation response, though it can’t prove an untested control works.
If ownership or test access is missing, assign it and manage the interim exposure. Our guide on how to prepare for an AI red teaming assessment covers the access and material testers need. An empty calendar is no reason to leave an unsafe operating state in place.
Work Through a Fictional AI Portfolio
This fictional portfolio illustrates how controls and evidence gaps lead to different actions. Each outcome depends on the stated assumptions.
| System and assumed state | Control evidence and gap | Decision and action |
|---|---|---|
| HR handbook Q&A. Employees only, approved non-sensitive corpus, no tools or external links. | Authorised editors control the corpus. Internal tests cover retrieval boundaries and misleading answers. Staff check consequential decisions against the handbook. No open material findings. | Continue bounded use with an HR owner, answer-quality checks, and an escalation route. Reassess if data, editing access, or intended use changes. No external slot. |
| Vendor CRM drafting feature. Reads customer records and inbound correspondence. A user reviews and sends each draft. | The vendor’s report covers the base CRM, with no evidence of tenant isolation or record-access enforcement for this feature. User review doesn’t show that drafts can’t disclose the wrong records. | Treat the scope as unknown. Restrict the pilot’s data and users, request feature-specific evidence before renewal, and establish which validation the vendor authorises. |
| Accounts-payable agent. Reads emailed invoices and supplier data, then creates payment proposals. Representative non-production environment; rollout in six weeks. | It can’t execute payments or edit supplier master data. A separate finance workflow requires an authorised approver to check payee and bank details against trusted records. Functional tests exist, but adaptive invoice manipulation and approval-boundary bypass remain untested. The team lacks specialist capacity for that work. | First candidate for the external slot. Validate proposal integrity and approval enforcement before rollout, and confirm the assessor’s scope fits the workflow. |
| Coding agent in use. Reads external issues. A broad CI credential permits production writes. | The team has confirmed the credential’s excessive scope. Reviewing the agent’s proposed code doesn’t prevent credential misuse. No review is scheduled. | Revoke or narrow the credential and restrict the agent now. Preserve evidence, investigate suspicious activity, assign an owner, and validate the revised boundary. |
| Loan pre-screening assistant. Controlled pilot that summarises applicant documents for a human credit reviewer, with no decision authority. | No evidence shows whether adversarial documents can distort extracted income or debt figures. The reviewer has no documented source-check procedure. | Investigate the extraction and review path before expansion. Introduce source verification and test it, internally if you have the expertise, or through a scoped assessment if you don’t. |
Accounts payable is the first external candidate because its rollout decision in six weeks needs evidence the internal team cannot provide. Test access and a representative baseline are available. The loan pilot can remain restricted while the team investigates its review controls; accounts payable needs assessment before rollout.
The coding agent needs the first security action because its excessive credentials are already in use. After restricting them, compare its remaining evidence gaps with accounts payable before allocating the available slot. It may displace that assessment if delay would leave a more consequential operating decision unsupported.
The vendor feature needs investigation because its feature-specific controls remain unverified. Direct testing requires authorised access and an agreed scope.
The loan assistant’s security case rests on distorted extraction influencing a credit review. Intended use in evaluating natural persons’ creditworthiness may bring it within Annex III, point 5(b), subject to the applicable classification rules. Its intended purpose informs classification; your provider or deployer role determines which obligations apply to your organisation.
Revisit the Choice When Its Assumptions Change
Re-rank a system after new permissions, tools, data sources, users, or intended uses, and after a model or provider change, a control failure, an incident, a near-miss, or a new assurance request. Track shared dependencies, because one change to a common service account can alter several inventory rows. A fast-changing system may need a regression baseline, but a high change rate doesn’t justify indefinite deferral. Give deferred work a review date as well as event triggers.
Record the Decision and Prepare for Scoping
Keep a compact record of the choice, whichever delivery route you take:
| Record field | What to retain |
|---|---|
| System and state | Named workflow, intended use, configuration reference, and review date |
| Action and rationale | Chosen route, consequential path, evidence gap, and reason for priority or deferral |
| Accountability | Action owner and the person authorised to accept interim exposure |
| Required evidence | The control or claim to validate, the relevant scenario, and the decision it supports |
| Timing and interim measures | Decision date or urgency, temporary restrictions, and delivery date |
| Reconsideration | Review date, change triggers, and open unknowns |
The next question is what evidence an assessment will give you. Explore the Provion platform and book a demo to see how assessment scope, test results, and the configuration behind each run come together for review. Use the walkthrough to discuss the evidence your team needs for its security decision.
If you’ve chosen the system and named the decision it must support, book a scoping call. Bring your decision record, an architecture sketch, and existing evidence to define the assessment scope and confirm whether Provion can support it.
