Image prompt injection, AI feature risk, and what security teams should be doing about it.
We’ve been curious about prompt injection for a while. Not in the abstract, theoretical sense, but in the “let’s actually build the attack and see what happens” sense. So we did.
In September 2025, we published a technical breakdown of the attack. This post is about the broader implications for security teams evaluating AI tools.
The setup was simple. We took the kind of image you would find in any repository, like a logo or a screenshot, and embedded malicious instructions inside it as text using PyRIT’s AddImageTextConverter. Then we dropped it into a git repository alongside normal project files and checked whether Claude Code would follow those instructions when a developer ran a routine query. Something like “analyze the files in this directory,” the kind of thing developers do constantly, without giving it a second thought.
It worked. Claude Code read the image, interpreted the rendered text as instructions within its LLM context, and chose to run the suggested commands. No exploit. No CVE. No vulnerability in the traditional sense, although prompt injection is increasingly recognized as its own vulnerability class (it’s LLM01 in the OWASP Top 10 for LLMs). This is an AI tool doing what AI tools do, processing input, and an attacker controlling what that input says.
You can read the full technical breakdown, including the PyRIT pipeline we used to automate and score the testing, in our AI Red Teaming Part 3 post.
This isn’t a story about Claude Code being an insecure product. Claude Code has meaningful security mitigations in place. It’s about a default configuration that hadn’t anticipated this specific attack vector, which is a much more common situation than outright negligence, and arguably a more important one to understand.
In our testing, the injected instructions executed 'whoami' and ifconfig without the user being prompted for permission. Both fall within Claude Code’s default allowed command set — read-only operations that the tool considers safe to run without explicit approval. It’s worth noting that ifconfig is allowed by default, while the newer ip commands require approval. This is the kind of inconsistency worth auditing in your own deployment. We also tested commands that would require user approval and, to Claude Code’s credit, those were blocked or prompted correctly.
So, from a default installation, the immediate impact is information disclosure: system user context and network configuration, triggered by something as innocent as analyzing a repository. That alone is worth knowing about. But the risk compounds from there. Combine it with a user environment that has broader permissions configured, or with social engineering techniques like Lies-In-The-Loop that trick a user into approving additional commands, and you’re looking at something considerably worse.
The only requirement for the attack to work was for the victim’s Claude Code session to access the poisoned file. That’s a low bar.
For teams using Claude Code or evaluating it for broader deployment, the mitigations exist and they work within their defined scope. The gap is in default configurations not being designed with adversarial image input in mind. That’s worth knowing, and worth testing in your specific environment before assuming the defaults are sufficient.
Prompt injection via image input isn’t a Claude Code problem. It’s a problem for any AI system that processes user-controlled content without adequate input validation, and that describes a rapidly growing category of tools.
Think about what’s getting AI features right now: code assistants, document summarization tools, customer support chatbots, and AI agents that send emails or query internal systems on a user’s behalf. Each of these has an input surface. Each of them, to varying degrees, follows instructions found in that input. Most of them haven’t been adversarially tested for prompt injection specifically.
To put this into context, imagine a contractor submits a pull request that includes a logo image with injection text rendered onto it. A developer on your team runs Claude Code to review the PR. The image gets processed, the instructions execute, and your developer never saw anything suspicious — they were just reviewing code. That’s the scenario. It doesn’t require a sophisticated attacker. It requires someone who has read this blog post.
We tested Claude Code because it’s a tool we use and wanted to understand it better. The methodology of using PyRIT to build a testing pipeline, automate prompt variants, and score results systematically can be directly applied to other AI targets.
One practical consequence of AI features shipping fast is that standard vendor security assessments don’t necessarily cover them well. A SOC 2 report covers the vendor’s infrastructure. It doesn’t say anything about what their AI feature does with your data once it leaves that infrastructure. Third-party model providers, training data policies, and prompt injection testing don’t show up in the standard assessment. You have to specifically ask for it.
These questions are worth adding to your standard vendor questionnaire now, before the next procurement cycle:
Most vendors won’t have polished answers to all of these yet. That’s fine. The goal right now is to start building the baseline and understand where the gaps are, not to disqualify vendors who are still figuring it out.
The right starting point depends on where your team is. There’s no single tool that works for everyone. If you’re just getting started, here’s what we would recommend:
Start manual. Open the AI tool, try to inject an instruction into whatever input it accepts, and see if it follows it. No setup, no tooling, immediate feedback. Try this before anything else.
Second, read the OWASP Top 10 for LLMs. It’s the clearest map of AI-specific risk categories available, written for security practitioners rather than AI researchers. It’ll give you language for what you’re finding.
Third, try PyRIT when manual testing starts feeling repetitive. It’s Microsoft’s open source AI red teaming framework that handles automation so you can focus on the interesting parts. Our Part 1 post walks through the setup and a first challenge end-to-end. It has a learning curve, but it’s worth it once you’re past the basics.
Finally, look at Garak. Different tool, different approach to probing model behavior. Some practitioners prefer it; at the very least, it’s worth knowing it exists.
Treat AI input surfaces the same way you’d treat any other user-facing input: assume someone will eventually try to abuse it, understand what the system can do when it follows those instructions, and test whether your controls would catch it. That framing applies regardless of which tools you use to get there.
First, figure out what the AI tool can actually do. What can it run, read, or reach without asking for permission? That’s your starting point. Anything your team does not need is a risk you can cut by changing the configuration.
Second, think about what ends up in your working directories. The attack we ran worked because one poisoned file was sitting where Claude Code was told to look. If your team uses AI tools to analyze code repositories that pull in third-party dependencies or external content, that’s a similar setup.
Third, check the default allowed command set and verify the approval prompts actually fire the way they’re supposed to. Default installations behave one way; customized ones sometimes don’t. Test your actual environment, not the documentation.
None of this requires a dedicated project. Most of it is an afternoon of configuration review. The point is to do it before someone else does it for you.
Start with whatever can do the most damage if it goes wrong, especially if someone else is controlling the instructions. We’re talking about tools that can take action: run commands, write files, call APIs, and send communications.
Then move to tools that can surface sensitive data even if they can’t act on it, such as document summarization, internal search, and customer-facing chatbots with account access. Lower urgency, still worth finding.
You can save the read-only, no-backend tools for last. They’re not zero risk, but they’re not where you need to start.
Within each tier, lean toward newer implementations first. Fresh AI features get less scrutiny than products that have been in production for years. That’s usually where the gaps are.
We want to be clear that the point of publishing this isn’t to claim Claude Code is broken. It isn’t. The point is that even well-built tools have edge cases that weren’t anticipated in their default configurations, and the only way to find those edge cases is to look for them.
That’s true of the AI tools you’re evaluating for procurement. It’s true of the ones already running in your environment. And it’s probably true of tools your staff is using that you don’t have visibility into yet.
The risk doesn’t show up in a vulnerability scanner. It shows up when someone with the right methodology goes looking for it. We’d rather that someone be you.
Our full PyRIT series, including setup, methodology, and real targets — is on the BreakPoint Labs blog. If you want to talk through AI testing scope for your environment, get in touch.
Prompt injection is an attack where a malicious actor embeds instructions into content that an AI model processes. When the model reads that content, it interprets the embedded instructions as legitimate commands and follows them. It’s classified as LLM01 in the OWASP Top 10 for Large Language Models
Image prompt injection is a variation of the attack where malicious instructions are embedded as visible text within an image file. When an AI tool processes the image, it reads the rendered text as part of its input and may run the instructions without user awareness.
Claude Code has security mitigations in place and handles many attack scenarios correctly. In our testing, a default installation didn’t anticipate adversarial image input as an attack vector. Injected commands that fell within the default allowed command set ran without prompting the user. Commands requiring explicit approval were blocked correctly. The risk is in default configurations, not a fundamental flaw in the product.
No. The attack requires embedding text into an image file and placing it in a location the AI tool will access. No exploit, no CVE, and no advanced technical skill are required beyond understanding how the tool processes input.
Audit what your AI tools can do without user approval. Review what files and directories those tools access. Verify that approval prompts fire correctly in your specific environment, not just in vendor documentation. Test your actual deployment, not the default installation.
Start manually. Open the tool, inject an instruction into whatever input it accepts, and observe the result. When manual testing feels repetitive, try PyRIT, Microsoft’s open source AI red teaming framework. It automates prompt variant generation and scores results at scale. It has a learning curve, but it is worth the investment once you’re past the setup. The OWASP Top 10 for LLMs is the best starting reference for understanding the full risk category.
Context discovery is a critical part of an in-depth application testing methodology. This is an area where loads of risk can be uncovered that web application scanners and past security assessments might have missed. At BreakPoint Labs (BPL), we place a strong focus on application content discovery. This blog post will share our approach to content discovery, with the hope of inspiring others to dig deeper in their next application security assessment.
If your content discovery workflow is just pointing ffuf or gobuster at a target with a giant generic wordlist, you’re probably leaving a massive amount of attack surface on the table. While there is nothing inherently wrong with these tools or using large generic wordlists, there are better ways.
Oftentimes, we get in a “one size fits all” mindset when it comes to specific workflows in our methodology. Run the same tools and syntax for each engagement, hoping to find everything in a single easy command. But think about it, if we are scanning a web application for unlinked content, do we think we are going to find all those hidden endpoints with one list? Doubtful.
A smarter approach to unlinked content is to build it in stages. Focus on the target. What tech stack is running on the application? What endpoints have we already discovered? What endpoints or words/patterns do we already know about? These are all questions we should ask ourselves when picking wordlists, and remember, this doesn’t have to be done in one large sweep. In fact, it is often better to do this in stages, analyzing what you have found (and not found) to better craft your next scan.
Lastly, consider the tools we are using for this. These content discovery tools are a dime a dozen. Ffuf, gobuster, dirsearch, feroxbuster, the list goes on. Nothing wrong with these tools, but they all basically do the same thing. So really, it’s up to you which one you like. If it works for you, use that. I typically stay within Burp Suite, using the Discover Content feature and sometimes Intruder. My main selling point here is simple: I can easily add what I discover to the site map, passively analyze the endpoints I discover for new linked content, and configure Burp Suite to leverage what it finds to build custom approaches for the specific target.
The tool can be accessed by right-clicking a host in your site map and going to Engagement Tools -> Discover Content. You will see three (3) tabs: Control, Config, and Site Map.
Let’s dive into the Config tab, where we can really fine-tune the scans we run. While I won’t go into every possible configuration, I will cover each unique setting, what they do, and the specific use cases I like to use.
The Target section is where we can, of course choose the target, change the discover types (Files and/or Directories), and a recursion limit. I typically leave this as is, depending on the target, so not much to talk about here.
The Filenames section focuses on which wordlist(s) to use and how to configure them.
The File extensions section is helpful for controlling the addition of various file extensions to file stems. When Burp tests for a file, it will also check for these additional file extensions configured in this section.
Lastly, the Discovery Engine section is used to control how the scan is used for things such as crawling the discovered content and interactions with the site map.
So far, we have just talked about how we can configure this tool to dynamically add extensibility to our content discovery. Now let’s talk about authentication. Most tools, like ffuf, at least to my knowledge, don’t have a great way to handle this. You can pass in authentication headers and cookies into these tools, but when the session expires, there is no automated way to handle it.
This is another perk of using Burp Suite’s Discover Content tool: it can be used with Burp’s built-in session handling and cookie jar features to automate this, making authenticated discovery easier. We won’t go into how to set up session handling rules, macros, or the cookie jar, as there are plenty of resources available for this. However, when using these features with the Discover Content tool, be sure to update the Use cookies from Burp’s cookie jar rule to include both the Target and Scanner tools. This will ensure your scan uses the cookie(s) in place there.
Now let’s talk about wordlists. What to use, when to use it, how to use it, and why it even matters. I see a lot of times people do one of two things: they either use a large, generic, publicly available wordlist that just hits on all cylinders. Or, they build custom wordlists that, well, are just collections of other wordlists that again hit on all cylinders. Don’t get me wrong, this, in and of itself, is not a bad idea. The problem is this is where they stop, but for me, this is just the beginning.
This is stage one; these are generic wordlists I like to use to see what’s out there. These are the classic wordlists we have all seen used. But they can get us a good idea of what is going on with the target and, in some cases, find easy low-hanging fruit. A few examples I like to use for this are:
Once you have completed a thorough walk-through of the application, maybe even crawled the site and conducted some unlinked content discovery using the lists above, it’s time to fine-tune your next phase.
I like to start by looking at the site map, noting all extensions I see, headers that can help identify the tech being used, interesting endpoints such as /api, /admin, /wp-content, etc. Also, don’t forget to look at things like Wapalyzer and passive collection notes in Burp to find additional tech you may not be tracking.
Once this data has been collected, it’s time to plan the next phase. Find wordlists that are specific to the application. For example, if the site is running WordPress, then use a WordPress-specific wordlist. If the site has a lot of php, then find wordlists targeting php endpoints, etc. This seems basic, but a lot of times people don’t take the time to really plan out this phase, and instead just try to throw everything at once and move on.
I’ll give a few examples I like to use here, but keep in mind there are plenty more out there. Do your research and find what you like.
Last tip I will give here is, be sure to also use custom wordlists. Starting with public wordlists like this is a great place to start, but there can be real power in using custom wordlists, not only for specific targets using tools like GAP and backup-gen, but for your own collection of private wordlists that can be used for other engagements. When you find endpoints or filenames that look interesting for a site you are targeting, be sure to add them to your wordlists wherever you find the most valuable, such as php files to your php-specific wordlists. Never know when you may find it again.
Another valuable thing when you have a large scope of many applications on a target domain is to use words, parameters, etc, from the other sites on the site you are focusing on. These sites could share similar patterns depending on the developers. For example, if you have dev sites in your scope, you may find that they use various parameters that are used in dev but that have made it into production. So be sure to grab these with GAP and try them on your site.
Let’s run through a quick example, putting all this information together to show how I typically approach new applications to test.
In summary, Burp Suite’s Discover Content tool, when configured properly, can greatly improve content discovery and, in my experience, find more content than other fuzzers. I’m not saying other tools are bad, nor that they don’t provide functionality that beats this tool. However, I wanted to take this opportunity to show the power of understanding your toolkit and leveraging all the features it has to offer to improve your workflow. If you already have a tool like this built into your methodology and don’t want to change, I highly encourage you to explore all the features your tool offers. Test out different configurations and wordlists, find what works best for you. In the end, the tool you use is subjective, but I hope this article helps you improve how you approach content discovery.
Trevor Isenberg is a Senior Red Team Operator at BreakPoint Labs for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. Coming from a background in system administration, Trevor provides a unique perspective when targeting complex networks and solving challenges. He holds numerous professional certifications and is especially interested in Web Application, Active Directory and ICS/SCADA pentesting. When he’s not in the middle of a red team engagment, you’ll find him spending time with family, playing guitar and working on projects at night to continuing growing his skillset.
Red teaming of AI-enabled systems is becoming essential for every organization to ensure they aren’t introducing risk to themselves and their stakeholders as they implement generative AI applications. Pentesters and red teamers need to become familiar with AI-specific attack techniques and terminology.
Microsoft has put out some AI Red-Teaming Playground Labs that are a great way to practice attacking AI applications. In this series of blog posts, we’ll walk through some of these challenges and uncover AI red-teaming tactics, techniques, and procedures (TTPs) that you can add to your arsenal.
The setup is a bit of effort initially, but I promise it’s worth it and won’t take much once it’s running. We’ll create an Azure account if you don’t already have one, set up an AI deployment for the Playground to use, then install and connect the Playground itself. Finally, effective offensive security testing should combine both automated and manual testing, so we’ll set up Microsoft’s PyRIT framework so we can walk through each challenge both manually and programmatically.
If you don’t have an Azure account, let’s set one up real quick. Cloud stuff always feels intentionally obscure and overly-complicated, so I’ll try to be detailed, but don’t blame me if this part is a pain. Also be aware of costs – new Azure accounts get $200 in free credits, and for this setup and Challenge 1 testing I’ve only used $0.09 of that, so no big deal here but just know that billing is happening in the background.
Here are the steps I took:
If all that worked successfully, congrats because it took me an embarrassingly long time to get to this point on the first pass. Let’s continue:
Success! Now you’ve got two models for the AI Red-Teaming Playground Labs to communicate with.
Install docker compose if you don’t have it already – if you’re using Kali like me, remember do not use apt install docker-compose! Do this instead:
apt install -y docker.io
mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose`
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose versionThis is a great time to mention that docker is my archnemesis and it always causes me personal pain to recommend using it, but here we are.
Next is setting up the labs locally. Pull down the git repo:
git clone https://github.com/microsoft/AI-Red-Teaming-Playground-Labs.gitThe repo has a .env.example file in it. Copy this to .env and follow the instructions inside it to create a SECRET_KEY and an AUTH_KEY, then populate the rest with your Azure endpoint and API key saved previously. Set the model as “gpt-4o”.
And that should be it! Within that directory, run:
docker compose upThen navigate to http://localhost:5000/login?auth=<AUTH_KEY> and you should see a list of challenges to launch.
Finally, we’ll install PyRIT, which is Microsoft’s open-source framework for automating security testing of generative AI applications. Its documentation can also be obscure and needs a little more detail in my opinion, but we’ll do our best to figure it out.
Install pyrit, jupyter, and ipykernel:
pip install pyrit jupyter ipykernelInstall kernelspec:
python -m ipykernel install --user --name=pyrit_kernelSetup complete!
Challenge 1 – Direct Prompt Injection is straightforward and simple enough to do manually. The exact same prompt won’t work 100% of the time due to the nature of GenAI being non-deterministic and all that, but you shouldn’t have much of a problem convincing the chat bot to give up its passwords.txt.
I recommend poking around the PyRIT documentation and watching at least this high-level walkthrough from Microsoft. Even better, watch all 10 quick episodes of their AI Red-Teaming 101 course.
At a high-level, here’s what’s happening in PyRIT:
Datasets = Initial prompts to be fed into the pipeline.
Orchestrator = from their docs: “The Orchestrator is a top-level component that red team operators will interact with the most. It is responsible for telling PyRIT which endpoints to connect to and how to send prompts. It can be thought of as the component that executes an attack technique.”
Converters = converts the prompts into something else – it could be different encoding, putting prompts into a Word doc instead of just plain text, or much more complex things like the MathPromptConverter which is used to “transform user queries into symbolic mathematical problems by applying set theory, abstract algebra, and symbolic logic”. We’ll look at a few of these in a second.
Target = an AI target and how to connect to it. For this it will be an HTTPTarget but other examples include OpenAIChatTarget, AzureBlobStorageTarget, and HuggingFaceChatTarget.
ScoringEngine = to avoid manually reviewing every chat response, you can set up scorers that will automatically grade responses to see if they meet set criteria. We won’t do this today but you’ll want this when doing high-volume testing.
Ok let’s launch the Jupyter notebook. For some of these labs, Microsoft has included a notebook to start with for PyRIT testing.
jupyter notebookWhen the web GUI launches, select File -> New -> Notebook and choose the pyrit_kernel when prompted for Kernel. On the right, Upload and choose “AI-Red-Teaming-Playground-Labs/notebooks/Lab 1 – Credential exfiltraiton.ipynb”.
Follow the instructions to fill in the raw_http_request variable – don’t forget to adjust the curly braces according to step 7, and then put your prompt variable into the “input” parameter as shown below. Run the first cell and double check the print output looks as expected.
Note: if you ever get a JSONDecode error running the second cell (where it makes requests to the Lab 1 challenge HTTP endpoint) it’s likely due to a 401 response because the session cookie expired – you’ll need a new cookie/request here.
Note 2: if you see a duckdb error about unhashable type, this is a known bug in the newest duckdb (1.40), revert to 1.3.2 with pip install "duckdb==1.3.2"
Now that we’ve got a valid POST request that will send prompts to the Challenge 1 endpoint, let’s look into Converters to see how the initial prompt can be manipulated. Since I want to test out a bunch of Converters, in the first cell I changed the prompt converter import statement to get all of them:
from pyrit.prompt_converter import *In the second cell, when it defines the HTTPTarget, add the parameter “use_tls=False” at the end or else you’ll get SSL errors (as the challenges are HTTP only). It should now look like:
http_prompt_target = HTTPTarget(http_request=raw_http_request, callback_function=parsing_function, timeout=20.0, use_tls=False)CharacterSpaceConverter is a simple option to test first. It does exactly as advertised – puts a space between every character. Within the orchestrator, set the prompt_converter to CharacterSpaceConverter and run the notebook – you should see the converted prompt and the chat bot’s response.
Within the repo, you can find a list of all the built-in converters inside “PyRIT/pyrit/prompt_converter/__init__.py” (at the time of testing, I counted 61 total). Play around with different converters to see what options you have. You can stack multiple Converters as well and they’ll execute sequentially – here’s Leetspeak followed by CharacterSpace:
For a more complicated Converter, I tried MathPromptConverter, which frames the prompt as advanced math logic to see if it will complete the “math problem” which might get you past security controls. This and some other Converters require an LLM resource that PyRIT can use to generate advanced prompt mutations (and yes, you’re using GenAI to generate attacks against GenAI, this is life now).
Fortunately we just set up an Azure OpenAI endpoint for the Playground Labs and can re-use that same resource. You will need to define the endpoint details (found in Azure AI Foundry -> Deployment -> gpt-4o) within the code or in a .env file as explained in the PyRIT documentation.
We’ll do it directly in the code for now. Add an OpenAIChatTarget for this out-of-band prompt generation and give it the model name, API key, and endpoint. Within the orchestrator function call, add the MathPromptConverter and point it to the OpenAIChatTarget you just defined. The code should look similar to below (I’ve redacted my endpoint details):
Take a second to thank the universe you didn’t have to dig up old high school calculus textbooks to craft this prompt. Unfortunately here, the Challenge 1 chat bot didn’t fall for the math trap.
Playing with the Converters was interesting and there are plenty of techniques to try. The first to succeed in retrieving passwords.txt was the PersuasionConverter using the option “expert_endorsement”:
And that’s it for Challenge 1! Despite the simple example, it’s easy to see the potential here for automating large scale testing of AI implementations. We’ll be back soon for Challenge 2 – Metaprompt Extraction.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.
