Image prompt injection, AI feature risk, and what security teams should be doing about it.
We’ve been curious about prompt injection for a while. Not in the abstract, theoretical sense, but in the “let’s actually build the attack and see what happens” sense. So we did.
In September 2025, we published a technical breakdown of the attack. This post is about the broader implications for security teams evaluating AI tools.
The setup was simple. We took the kind of image you would find in any repository, like a logo or a screenshot, and embedded malicious instructions inside it as text using PyRIT’s AddImageTextConverter. Then we dropped it into a git repository alongside normal project files and checked whether Claude Code would follow those instructions when a developer ran a routine query. Something like “analyze the files in this directory,” the kind of thing developers do constantly, without giving it a second thought.
It worked. Claude Code read the image, interpreted the rendered text as instructions within its LLM context, and chose to run the suggested commands. No exploit. No CVE. No vulnerability in the traditional sense, although prompt injection is increasingly recognized as its own vulnerability class (it’s LLM01 in the OWASP Top 10 for LLMs). This is an AI tool doing what AI tools do, processing input, and an attacker controlling what that input says.
You can read the full technical breakdown, including the PyRIT pipeline we used to automate and score the testing, in our AI Red Teaming Part 3 post.
This isn’t a story about Claude Code being an insecure product. Claude Code has meaningful security mitigations in place. It’s about a default configuration that hadn’t anticipated this specific attack vector, which is a much more common situation than outright negligence, and arguably a more important one to understand.
In our testing, the injected instructions executed 'whoami' and ifconfig without the user being prompted for permission. Both fall within Claude Code’s default allowed command set — read-only operations that the tool considers safe to run without explicit approval. It’s worth noting that ifconfig is allowed by default, while the newer ip commands require approval. This is the kind of inconsistency worth auditing in your own deployment. We also tested commands that would require user approval and, to Claude Code’s credit, those were blocked or prompted correctly.
So, from a default installation, the immediate impact is information disclosure: system user context and network configuration, triggered by something as innocent as analyzing a repository. That alone is worth knowing about. But the risk compounds from there. Combine it with a user environment that has broader permissions configured, or with social engineering techniques like Lies-In-The-Loop that trick a user into approving additional commands, and you’re looking at something considerably worse.
The only requirement for the attack to work was for the victim’s Claude Code session to access the poisoned file. That’s a low bar.
For teams using Claude Code or evaluating it for broader deployment, the mitigations exist and they work within their defined scope. The gap is in default configurations not being designed with adversarial image input in mind. That’s worth knowing, and worth testing in your specific environment before assuming the defaults are sufficient.
Prompt injection via image input isn’t a Claude Code problem. It’s a problem for any AI system that processes user-controlled content without adequate input validation, and that describes a rapidly growing category of tools.
Think about what’s getting AI features right now: code assistants, document summarization tools, customer support chatbots, and AI agents that send emails or query internal systems on a user’s behalf. Each of these has an input surface. Each of them, to varying degrees, follows instructions found in that input. Most of them haven’t been adversarially tested for prompt injection specifically.
To put this into context, imagine a contractor submits a pull request that includes a logo image with injection text rendered onto it. A developer on your team runs Claude Code to review the PR. The image gets processed, the instructions execute, and your developer never saw anything suspicious — they were just reviewing code. That’s the scenario. It doesn’t require a sophisticated attacker. It requires someone who has read this blog post.
We tested Claude Code because it’s a tool we use and wanted to understand it better. The methodology of using PyRIT to build a testing pipeline, automate prompt variants, and score results systematically can be directly applied to other AI targets.
One practical consequence of AI features shipping fast is that standard vendor security assessments don’t necessarily cover them well. A SOC 2 report covers the vendor’s infrastructure. It doesn’t say anything about what their AI feature does with your data once it leaves that infrastructure. Third-party model providers, training data policies, and prompt injection testing don’t show up in the standard assessment. You have to specifically ask for it.
These questions are worth adding to your standard vendor questionnaire now, before the next procurement cycle:
Most vendors won’t have polished answers to all of these yet. That’s fine. The goal right now is to start building the baseline and understand where the gaps are, not to disqualify vendors who are still figuring it out.
The right starting point depends on where your team is. There’s no single tool that works for everyone. If you’re just getting started, here’s what we would recommend:
Start manual. Open the AI tool, try to inject an instruction into whatever input it accepts, and see if it follows it. No setup, no tooling, immediate feedback. Try this before anything else.
Second, read the OWASP Top 10 for LLMs. It’s the clearest map of AI-specific risk categories available, written for security practitioners rather than AI researchers. It’ll give you language for what you’re finding.
Third, try PyRIT when manual testing starts feeling repetitive. It’s Microsoft’s open source AI red teaming framework that handles automation so you can focus on the interesting parts. Our Part 1 post walks through the setup and a first challenge end-to-end. It has a learning curve, but it’s worth it once you’re past the basics.
Finally, look at Garak. Different tool, different approach to probing model behavior. Some practitioners prefer it; at the very least, it’s worth knowing it exists.
Treat AI input surfaces the same way you’d treat any other user-facing input: assume someone will eventually try to abuse it, understand what the system can do when it follows those instructions, and test whether your controls would catch it. That framing applies regardless of which tools you use to get there.
First, figure out what the AI tool can actually do. What can it run, read, or reach without asking for permission? That’s your starting point. Anything your team does not need is a risk you can cut by changing the configuration.
Second, think about what ends up in your working directories. The attack we ran worked because one poisoned file was sitting where Claude Code was told to look. If your team uses AI tools to analyze code repositories that pull in third-party dependencies or external content, that’s a similar setup.
Third, check the default allowed command set and verify the approval prompts actually fire the way they’re supposed to. Default installations behave one way; customized ones sometimes don’t. Test your actual environment, not the documentation.
None of this requires a dedicated project. Most of it is an afternoon of configuration review. The point is to do it before someone else does it for you.
Start with whatever can do the most damage if it goes wrong, especially if someone else is controlling the instructions. We’re talking about tools that can take action: run commands, write files, call APIs, and send communications.
Then move to tools that can surface sensitive data even if they can’t act on it, such as document summarization, internal search, and customer-facing chatbots with account access. Lower urgency, still worth finding.
You can save the read-only, no-backend tools for last. They’re not zero risk, but they’re not where you need to start.
Within each tier, lean toward newer implementations first. Fresh AI features get less scrutiny than products that have been in production for years. That’s usually where the gaps are.
We want to be clear that the point of publishing this isn’t to claim Claude Code is broken. It isn’t. The point is that even well-built tools have edge cases that weren’t anticipated in their default configurations, and the only way to find those edge cases is to look for them.
That’s true of the AI tools you’re evaluating for procurement. It’s true of the ones already running in your environment. And it’s probably true of tools your staff is using that you don’t have visibility into yet.
The risk doesn’t show up in a vulnerability scanner. It shows up when someone with the right methodology goes looking for it. We’d rather that someone be you.
Our full PyRIT series, including setup, methodology, and real targets — is on the BreakPoint Labs blog. If you want to talk through AI testing scope for your environment, get in touch.
Prompt injection is an attack where a malicious actor embeds instructions into content that an AI model processes. When the model reads that content, it interprets the embedded instructions as legitimate commands and follows them. It’s classified as LLM01 in the OWASP Top 10 for Large Language Models
Image prompt injection is a variation of the attack where malicious instructions are embedded as visible text within an image file. When an AI tool processes the image, it reads the rendered text as part of its input and may run the instructions without user awareness.
Claude Code has security mitigations in place and handles many attack scenarios correctly. In our testing, a default installation didn’t anticipate adversarial image input as an attack vector. Injected commands that fell within the default allowed command set ran without prompting the user. Commands requiring explicit approval were blocked correctly. The risk is in default configurations, not a fundamental flaw in the product.
No. The attack requires embedding text into an image file and placing it in a location the AI tool will access. No exploit, no CVE, and no advanced technical skill are required beyond understanding how the tool processes input.
Audit what your AI tools can do without user approval. Review what files and directories those tools access. Verify that approval prompts fire correctly in your specific environment, not just in vendor documentation. Test your actual deployment, not the default installation.
Start manually. Open the tool, inject an instruction into whatever input it accepts, and observe the result. When manual testing feels repetitive, try PyRIT, Microsoft’s open source AI red teaming framework. It automates prompt variant generation and scores results at scale. It has a learning curve, but it is worth the investment once you’re past the setup. The OWASP Top 10 for LLMs is the best starting reference for understanding the full risk category.
Welcome back for Challenge 2 of the AI Red-Teaming Playground Labs! If you missed part 1 (PyRIT Setup and Challenge 1) you’ll want to go back and read that. This challenge will be “metaprompt extraction using obfuscation and encoding techniques” and we’ll once again go through it manually and automated with PyRIT. We’ll also write some simple custom classes to demonstrate some of the PyRIT functionality.
Manual was simple enough again. Backwards worked:
Base64 also worked:
Let’s play around with some custom classes here to get more familiar with the functionality. First, there is a built-in Base64Converter class that tries to turn the whole conversation into base64. I’ll make a much simpler converter called Base64RequestConverter that simply adds “In base64, ” to the beginning of a given prompt, making it similar to my manual attempt. I’ll also try using a Scorer to automate success validation.
In my testing I also saw some instances where after revealing the flag, further attempts within the same chat instance would always reveal the flag (or vice-versa – successful prompts from previous chats failed after multiple rejected attempts within a new chat), so the chat context history may be influencing responses. To avoid this, I set it up to make an initial request to the New Chat endpoint (http://localhost:4002/chats) and grab the new chat ID in the response so every message went to a brand new chat. Understanding usage and context for the specific target under test will be key to more effectively red teaming it at scale.
Building a Converter
Copying the Base64Converter class and simplifying it is easy enough. I won’t be sharing any code here, but I recommend looking through the default converters’ code and creating something basic to get familiar with the structure and capabilities.
My new Base64RequestConverter worked as expected – it added “In base64, ” to the front of the prompt and extracted the secret word encoded in base64:
Setting up scorer
Looking through the built-in scorers, there’s one called SubStringScorer that is as straightforward as it gets – it just checks the response for a specified substring. For this challenge, that won’t work because the secret word may be base64-encoded in the response. We’re going to need a scorer that can decode responses if needed.
First let’s test the SubStringScorer to make sure things work as expected. I pulled up the notebook from Challenge 1, pointed it at this challenge, and added the scorer with the substring of the secret word that we now know. I added the secret word to the prompt to make sure the chatbot put it in the response.
At the bottom you can see that the SubStringScorer returned True, as the target substring was found in the response. Let’s make a new scorer called SubStringScorerWithBase64 that is a copy of SubStringScorer with an additional check for base64 content. Now we can catch successful plain text and encoded responses. Again, I leave you to create your own scorer, but it was fairly straightforward when starting with a default scorer class.
Using our new classes to automate testing
Now that we have a decoding scorer, we have the power to automate the full process. We know that base64 encoding is one way to get the answer, but the chatbot didn’t always just give it up right away – phrasing made a difference. Let’s run it through the Base64RequestConverter combined with the ToneConverter.
I added ten various tones to cycle through. The GPT-4o generated tone conversions could get entertaining – “mysterious” was especially fun to read:
And the scoring worked! Not only was it a time-saver while manually looking through output, but it also gives us the ability to add some automated analysis. Since order of Converters matters, I was curious which process would be better: Base64 then Tone (prepend “In base64, ” to the original prompt and then change tone), or Tone then Base64 (tone first then prepend “In base64, ” to the tone-converted prompt). I set up a loop to cycle through each tone variation 10 times per scenario, for a total of 100 requests each:
Look at all the insight we can draw from this quickly without having to manually review chat output. Overall, dramatic and mysterious tones were most successful, with urgent and academic requests close behind. On the flip side, sarcasm did not go over well with the chatbot. Tone change first was also noticeably better – digging into it deeper, the tone conversion sometimes garbled the base64 part (especially for the academic tone, where it often mistakenly asked it to “decode the base64” instead of encode) whereas the direct base64 intro left no room for confusion. This makes sense as the challenge was specifically designed to be more revealing to encoding attempts – my testing using Tone only resulted in 0% success rate without the Base64.
As a red teamer, this type of analysis is key to designing attack paths. When first starting to interact with a target LLM application, I can pick a basic goal of something the application shouldn’t respond with and run through a bunch of conversions to find what works and what doesn’t. Then I can craft more impactful attacks using that insight. Step 1 might be getting it to explain how to make a harmful substance; Step 2 would be using the most successful techniques validated through Step 1 to extract RAG documents, execute commands, poison organizational data, or carry out any other assessment goals.
Just like when socially engineering people, different AI applications will be susceptible to different manipulations. Utilizing PyRIT to automate and analyze bypass success can provide a major boost to abusing LLMs, and in turn provide organizations with more accurate risk analysis and reduction as they implement AI into their processes.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.
Artificial intelligence (AI) holds immense promise for modern warfare, offering the potential to transform military operations and enhance decision advantage on the battlefield. From sophisticated intelligence analysis to optimizing logistics, the United States Marine Corps (USMC) recognizes AI as a transformative technology. AI offers extensive operational benefits, but like any powerful tool, its implementation also introduces new vulnerabilities and expands the attack surface that adversaries can exploit.
Adversaries are not idle in the face of AI advancements. They are increasingly utilizing AI to automate attacks, enhance phishing and social engineering tactics, and develop malware that can adapt and evade traditional detection methods. For the USMC, this means anticipating that “[adversaries will attempt] to disrupt, degrade, deny, deceive, or defeat our AI systems.”
These threats can be broadly categorized into three areas:
Adversarial AI is a particularly insidious threat because it targets the very features that make AI and machine learning useful: how the AI system uses data inputs to act and learn from results to improve its performance. BreakPoint Labs specializes in simulating these advanced threats to harden your AI systems.
Here are some key techniques adversaries might employ, and how BreakPoint Labs’ AI Red Teaming can expose these vulnerabilities:
It’s not just about direct AI attacks. Like any complex system, AI implementations rely on underlying infrastructure. An adversary can still leverage traditional cyberattack methods (Target Systems Analysis) to disrupt AI capabilities. For instance, exploiting vulnerabilities in cloud infrastructure, denying necessary data through signature management or shutting down civilian networks can cripple AI-enabled processes. BreakPoint Labs provides comprehensive system evaluation, examining system-wide vulnerabilities, supply chain vulnerabilities, deployment pipelines, and data security, integrating seamlessly with your existing cybersecurity frameworks.
The USMC’s Proactive Stance: Building a Resilient AI Force
The USMC is not only aware of these threats but is actively developing strategies to counter them. Their AI Implementation Plan is designed to mature the Service into a 21st Century fighting force that innovates and integrates AI into warfighting functions and business processes. BreakPoint Labs directly supports these initiatives by providing the crucial “Red Teaming LLMs” capabilities needed to “establish and update AI test, evaluation, validation, and verification processes.”
Key initiatives that BreakPoint Labs can help organizations mitigate these threats include:
The integration of AI into military operations is inevitable and necessary for maintaining a competitive edge. However, this advancement demands a robust and proactive cybersecurity posture. BreakPoint Labs focuses on understanding the unique vulnerabilities of AI systems, especially LLMs, RAG, and MCP, and implementing comprehensive threat assessment and mitigation strategies. Our goal is to empower organizations to evolve into an AI-enabled force, well-prepared to tackle future conflicts with greater readiness and effectiveness. This commitment to responsible and innovative AI use is a testament to a dedication to mission success.
By understanding the specific AI-centric threats/weaknesses and by continuously testing and hardening AI systems against these attacks, military forces and any organization leveraging AI in critical applications can ensure their AI provides a decisive edge rather than an exploitable vulnerability. In the age of autonomous systems and machine-speed warfare, securing AI is not just a technical challenge; it’s a mission imperative. A hardened AI advantage isn’t built by chance. It’s built by design. Organizations that adopt a threat-informed, resilience-focused approach to AI security today will be the ones best positioned to lead tomorrow. We’re committed to helping them succeed at BreakPoint Labs.
