LLM Penetration Testing: A Step by Step Method
Key takeaways
- What it is. LLM penetration testing is a scoped, authorised test of an application built on a large language model, aimed at finding real security weaknesses.
- It is more than prompts. The model is one part. Tools, retrieval, output handling, identity and logging matter as much.
- Test by channel. Direct input, external content, retrieval, tools and output each need their own tests.
- Evidence beats opinion. A finding should rest on a real side effect, such as leaked data or an executed action, not on what the model said about itself.
- Map to OWASP. Report against the OWASP Top 10 for LLM Applications and, for agents, the agentic list.
Most companies now have at least one application with a language model behind it. A support bot, a coding assistant, an internal search tool. Few have tested it the way they would test a web application.
LLM penetration testing closes that gap. This guide gives a practical method you can follow, whether you are a pentester adding AI to your skills or a security lead scoping a first test.
What is LLM penetration testing?
It is a structured, authorised attempt to break an LLM powered application the way an attacker would, and to document what worked. The target is the whole application, which includes the model, its instructions, the data it can retrieve, the tools it can call, how its output is used, and the infrastructure around it.
The OWASP GenAI Red Teaming Guide, version 1.0, frames this around four areas. They are model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. OWASP also lists a GenAI Red Teaming Manual version 1.0 on its resources page, dated 20 October 2026. A useful method covers all four areas.
How is it different from AI red teaming?
The two terms overlap and people use them loosely. In our guide to what AI red teaming is, we describe the broader adversarial practice. Penetration testing is usually the narrower, more repeatable engagement. It has a defined scope, a time box, a checklist of attack classes and a report with findings and fixes. Red teaming is often more open ended and may test people and process as well.
In practice you can use the same skills for both. This post focuses on the structured test.
Step 1, agree scope and rules
Before you send a single prompt, write down what is in scope and what is not. Cover
- which application, model and environment you may test
- which tools and data sources are in scope, and which actions are off limits
- whether you may test with real data or only test data
- limits on volume, since attacks can run up model costs
- who to contact if you find something serious
Get written authorisation. Testing a third party model or hosted service without permission can breach its terms or the law.
Step 2, recon
Learn what you are attacking.
- What model and version sits behind it
- What does the system prompt appear to say
- Which tools can it call, and with what permissions
- Which data does it retrieve, and who controls that data
- How is its output used, for example rendered in a page or passed to another system
- Which users and roles exist
Much of this comes from talking to the system, reading documentation and asking the developers. Good recon turns a vague test into targeted attacks.
Step 3, build a threat model
Write down the worst outcomes, then work backwards. What would a leak of the knowledge base cost. What if the agent could be made to send email, change a record or run code. Rank the outcomes. You will spend your limited time on the ones that matter.
Step 4, test by attack channel
Group tests by how the payload reaches the model. This keeps coverage clear.
| Channel | Examples to test | OWASP LLM theme |
|---|---|---|
| Direct input | Prompt injection, jailbreaks, multi turn guardrail erosion, system prompt extraction | Prompt injection, system prompt leakage |
| External content | Instructions hidden in web pages, files, emails, images or audio | Prompt injection |
| Retrieval | Poisoned documents, access control gaps, membership inference | Vector and embedding weaknesses, sensitive information disclosure |
| Tools and plugins | Confused deputy flaws, tool definition poisoning, excessive permissions | Excessive agency, supply chain |
| Output handling | Cross site scripting, SQL injection, SSRF, code execution from model output, Markdown image exfiltration | Improper output handling |
| Multi agent flows | Injection into a planner that an executor then obeys | Excessive agency, agentic risks |
| Cost and availability | Prompts that force huge outputs or loops | Unbounded consumption |
Many of the serious findings come from the last mile. The model produced something harmless looking, and the application then treated it as trusted. A model that writes a SQL query or a URL for a downstream system turns a prompt injection into a classic web vulnerability.
Step 5, collect evidence you can trust
A model that says it was jailbroken is not evidence. Look for a real side effect.
- A secret or personal record appears in the output
- An action happens in a connected system
- A script executes, or a request reaches an internal address
- A restricted role gains access it should not have
This is also why we built Mirage to score on real, decoded side effects and never to let a model grade itself. Models can be talked into agreeing that an attack worked when it did not, and into missing ones that did.
Run each promising attack several times. Language models are probabilistic, so record how often it works as well as whether it works.
Step 6, rate and report
For each finding record
- what you did, with the exact input and the response
- what the impact is in business terms
- how reliably it reproduces
- which OWASP category it maps to
- a specific fix
Keep fixes practical. Typical ones are to move access checks out of the prompt and into the tool, remove unneeded tools, validate and encode model output before using it, restrict outbound destinations, and require approval for sensitive actions. Prompt wording changes alone are rarely a lasting fix.
Step 7, retest
Fix, then test again. Models, prompts and data change, so a clean result is a snapshot. Many teams now run a smaller set of attack tests on every release.
Tools to speed things up
Manual testing finds the clever bugs. Automation covers the wide, repeatable ground.
- Mirage is our open source browser extension. It runs an autonomous recon, plan, test, judge and iterate loop against an AI chat interface or an API endpoint, across direct, indirect and agentic attack classes, and exports an HTML or JSON report with OWASP mapping.
- FreakLabs is where to build the skill first. It has 20 browser based attack scenarios across four difficulty levels, from prompt injection leading to cross site scripting through to tool definition poisoning and multi agent chain attacks. Two labs are free.
Other open source projects exist for scanning and prompt injection testing, and the right choice depends on your stack. Whatever you use, check its recent maintenance and run it only against systems you are allowed to test.
A starter checklist
- Written scope and authorisation in place
- Recon notes on model, tools, data and roles
- Threat model with ranked worst outcomes
- Tests run for every channel in the table
- Evidence based on real side effects
- Findings mapped to OWASP with fixes
- Retest scheduled
Conclusion
If you have never tested an AI application, start small. Pick one low risk application you are allowed to test, write the scope down, and work through the channels in the table. Build the hands-on skill in FreakLabs first, then use Mirage to automate the repeatable tests.
Frequently asked questions
What is LLM penetration testing?
It is a structured, authorised attempt to break an application built on a large language model the way an attacker would, and to document what worked. The target is the whole application, including the model, its instructions, retrieval data, tools, output handling and surrounding infrastructure.
How is LLM penetration testing different from AI red teaming?
The terms overlap. Penetration testing is usually the narrower, more repeatable engagement with a defined scope, a time box, a checklist of attack classes and a report with fixes. Red teaming is often more open ended and may test people and process as well.
What should be in scope for an LLM pentest?
Write down which application, model and environment you may test, which tools and data sources are in scope, which actions are off limits, whether real data may be used, limits on volume because attacks can raise model costs, and who to contact about a serious finding. Get written authorisation first.
How do you prove an LLM vulnerability is real?
Look for a real side effect, not the model's own claim. Examples are a secret or personal record in the output, an action in a connected system, a script that executes or a request that reaches an internal address. Repeat each attack several times, because models are probabilistic.
Which frameworks should an LLM pentest report map to?
Map findings to the OWASP Top 10 for LLM Applications and, for agents, the OWASP Top 10 for Agentic Applications. The OWASP GenAI Red Teaming Guide also frames testing around model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis.