Updated
Test what an AI application can actually do
AI security tests need to follow data access and tool permissions through the application, with a clear definition of success.
An assistant can answer a question correctly and still use the wrong customer’s data to do it. It can follow a document’s instructions and send information somewhere the user never approved.
Those failures require tests that include the application’s data and tools. A web challenge suite built around SQL injection or template injection doesn’t automatically measure them.
Start with an unauthorized outcome
A prompt that changes the tone of an answer isn’t the same as a prompt that causes a cross-account read. Before testing, define the protected resource and the boundary an attacker would need to cross.
For a support assistant, that might mean retrieving another tenant’s ticket. For a coding agent, it might mean treating repository content as permission to execute an unrelated action. For an email assistant, it might mean sending a message because instructions appeared inside an email it was supposed to summarize.
Each case needs a controlled environment and an observable success condition. A model saying that it performed the action doesn’t establish that it did.
Follow the data into the tools
Indirect prompt injection places instructions in material the application processes: a retrieved document, webpage, message, or tool response. The security question is what happens when the assistant treats that material as authority.
Record the full path. What did the attacker control? How did the application ingest it? Which tool call followed? Did the tool enforce the user’s permissions independently?
Tool access alone doesn’t establish a vulnerability. A failed request to a correctly protected API can show that a boundary held, even when the assistant attempted the wrong action.
Keep different outcomes separate
System-prompt disclosure can reveal implementation details. It isn’t automatically a secret leak; assess what the disclosed material actually contains. Credentials shouldn’t depend on a model keeping them hidden in its instructions.
A jailbreak evaluation measures whether a model follows a behavior policy under adversarial prompting. An application-security evaluation may instead measure unauthorized access or actions. Both can be useful, but their success rates answer different questions.
Cross-user leakage also needs an actual shared-data path. Asking a chatbot about a previous user doesn’t prove that another user’s conversation was accessible. Generated names and invented records mustn’t count as retrieved data.
Combine repeatable cases with adaptive testing
Fixed regression tests make it possible to compare builds and catch known failures. Adaptive agents can explore variations, carry context across turns, and follow responses into new paths. Human review can resolve ambiguous outcomes and check whether the test represents a realistic threat.
For each approach, preserve the attempts and apply the same success rule. Count the clean controls too: a collection containing only exploitable targets cannot establish a false-positive rate.
Our early internal AI challenges helped us exercise this workflow. Their size and scope didn’t justify a broad claim about application coverage. The next useful test is the one that reflects a real permission or data boundary in the system being shipped.