Artificial intelligence (AI) holds immense promise for modern warfare, offering the potential to transform military operations and enhance decision advantage on the battlefield. From sophisticated intelligence analysis to optimizing logistics, the United States Marine Corps (USMC) recognizes AI as a transformative technology. AI offers extensive operational benefits, but like any powerful tool, its implementation also introduces new vulnerabilities and expands the attack surface that adversaries can exploit.
Adversaries are not idle in the face of AI advancements. They are increasingly utilizing AI to automate attacks, enhance phishing and social engineering tactics, and develop malware that can adapt and evade traditional detection methods. For the USMC, this means anticipating that “[adversaries will attempt] to disrupt, degrade, deny, deceive, or defeat our AI systems.”
These threats can be broadly categorized into three areas:
Adversarial AI is a particularly insidious threat because it targets the very features that make AI and machine learning useful: how the AI system uses data inputs to act and learn from results to improve its performance. BreakPoint Labs specializes in simulating these advanced threats to harden your AI systems.
Here are some key techniques adversaries might employ, and how BreakPoint Labs’ AI Red Teaming can expose these vulnerabilities:
It’s not just about direct AI attacks. Like any complex system, AI implementations rely on underlying infrastructure. An adversary can still leverage traditional cyberattack methods (Target Systems Analysis) to disrupt AI capabilities. For instance, exploiting vulnerabilities in cloud infrastructure, denying necessary data through signature management or shutting down civilian networks can cripple AI-enabled processes. BreakPoint Labs provides comprehensive system evaluation, examining system-wide vulnerabilities, supply chain vulnerabilities, deployment pipelines, and data security, integrating seamlessly with your existing cybersecurity frameworks.
The USMC’s Proactive Stance: Building a Resilient AI Force
The USMC is not only aware of these threats but is actively developing strategies to counter them. Their AI Implementation Plan is designed to mature the Service into a 21st Century fighting force that innovates and integrates AI into warfighting functions and business processes. BreakPoint Labs directly supports these initiatives by providing the crucial “Red Teaming LLMs” capabilities needed to “establish and update AI test, evaluation, validation, and verification processes.”
Key initiatives that BreakPoint Labs can help organizations mitigate these threats include:
The integration of AI into military operations is inevitable and necessary for maintaining a competitive edge. However, this advancement demands a robust and proactive cybersecurity posture. BreakPoint Labs focuses on understanding the unique vulnerabilities of AI systems, especially LLMs, RAG, and MCP, and implementing comprehensive threat assessment and mitigation strategies. Our goal is to empower organizations to evolve into an AI-enabled force, well-prepared to tackle future conflicts with greater readiness and effectiveness. This commitment to responsible and innovative AI use is a testament to a dedication to mission success.
By understanding the specific AI-centric threats/weaknesses and by continuously testing and hardening AI systems against these attacks, military forces and any organization leveraging AI in critical applications can ensure their AI provides a decisive edge rather than an exploitable vulnerability. In the age of autonomous systems and machine-speed warfare, securing AI is not just a technical challenge; it’s a mission imperative. A hardened AI advantage isn’t built by chance. It’s built by design. Organizations that adopt a threat-informed, resilience-focused approach to AI security today will be the ones best positioned to lead tomorrow. We’re committed to helping them succeed at BreakPoint Labs.
Artificial-intelligence workloads – whether a multimodal LLM chatbot, a computer-vision pipeline, or an autonomous decision engine – behave very differently from classic software. They ingest massive, often shifting data sets, invoke outside services, and keep learning as they run. Those traits open fresh attack surfaces, from prompt manipulation to model exfiltration, that traditional security testing rarely touches. BreakPoint Labs has refined a methodology for AI penetration testing and red teaming that exposes genuinely exploitable weaknesses in AI systems and, crucially, translates them into business-level risk that the customer can act on.
Any effective assessment begins by drawing the boundaries of the digital terrain. Through conversations with asset owners, we define every component to be assessed: model endpoints, plug-ins, vector or RAG stores, fine-tuning pipelines, orchestration agents, and any other AI-related technologies. Scoping this correctly at the outset ensures that components that might expose you to risk won’t slip through the cracks.
Next, we explore adversaries and likely attack paths that match the customer’s risk profile. A rogue insider with privileged MLOps access, a well-funded nation-state, a competing vendor armed only with the public API, or a hobbyist jailbreak enthusiast each brings different tactics and constraints. By making those assumptions explicit, we avoid the common mismatch in which testers probe vulnerabilities that an actual attacker would never reach, or miss the ones that keep CISOs awake at night.
Finally, we set measurable success criteria. A privacy-focused healthcare client may care most about coercing a model to leak protected health information, whereas a university might worry about its public-facing AI services being exploited to gain internal access.
Passive research often uncovers more than clients expect. Public model cards might reveal the date weights were last updated or hint at third-party data sources; GitHub commits can expose access tokens left in a forgotten notebook. Documentation from academic papers sometimes lists hyper-parameters that make gradient-inversion attacks easier.
Collaborative discovery sessions are usually recommended. Sitting down with MLOps and DevSecOps engineers, we trace data lineage from raw collection to fine-tuning to inference, noting where sensitive inputs flow back into training.
Active enumeration rounds out the picture. We explore what is accessible from various vantage points, system and application information that can be gathered unauthenticated, and any other details an attacker could use maliciously. This discovery paints a realistic picture of what a potential attacker sees and what their next steps might be.
Automation gives us breadth. We fire up PyRIT and garak to orchestrate prompt injection and jailbreak attacks, at a scale no human could match. For pure prompt mutation, FuzzLLM stitches together template-based attacks, rapidly testing thousands of structural variations to find novel jailbreaks that slip past naive guardrails. When the target includes vision, audio, or tabular ML components, IBM’s Adversarial Robustness Toolbox (ART) generates perturbations, poison samples, and extraction queries tailored to each modality.
Automated runs are not a scoreboard; they are a sensor network. Initial findings feed the queue for human follow-up and confirmation.
Automation sketches the outlines; skilled operators color them in. A prompt that convinces the model to echo a single internal variable, for instance, can be expanded into a multi-turn jailbreak that exfiltrates the entire system, prompt, API keys, or confidential user data. A minor encoding leak might escalate into full remote code execution if an LLM, equipped with function-calling, can be persuaded to write an arbitrary Bash script that the orchestrator helpfully runs on the host.
Manual work also excels at chained attacks. Suppose reconnaissance shows that user-generated PDFs are fed into a vector store for semantic search. We craft a malicious PDF whose text embeds an indirect prompt injection. Once ingested, the user simply asks a normal question; the retriever surfaces our payload, and the model executes instructions that bypass the front-end filter, sending the conversation history to an external URL. Because this crosses storage, parsing, embedding, retrieval, and inference layers, no single automated tool would exercise it end-to-end.
This targeted manual testing combines with attacker knowledge and customer information to accurately assess and define risk and possible impact. Standards like the OWASP LLM Top 10 can be applied to synthesize current security posture. Mapping exploits back to MITRE ATLAS and the traditional MITRE ATT&CK matrices helps security teams slot each finding into a familiar defensive playbook.
Checking boxes is not enough; customers need to see how an attacker would actually weave techniques into a campaign. We therefore design narrative scenarios specific to the customer and environment.
These scenarios help accurately define risk and assess defense-in-depth capabilities, as well as develop thorough attack surface understanding. Preparation here leads to better detection and response when the bad guys do come around.
The final report begins with an executive narrative, transitions into a high-level analysis of the tested components, and delves into each vulnerability, detailing what we did, what the model did, and why it matters. Most importantly, we include tailored fixes and provide free remediation testing once implemented.
We close with interactive out-briefs, where red-teamers and client engineers review key exploits, discuss mitigations, and assess risk and potential impact. Customers leave not only with a list of vulnerabilities but also with prioritized and practical fixes and improvements.
By the end of this process, organizations understand exactly how an adversary can penetrate, pivot, and profit from their AI stack. They receive fixes that align with their architecture and risk appetite, not generic best-practice slides. Most importantly, they gain the confidence to innovate with AI, knowing their systems have already survived the sort of focused, creative abuse that attackers are perfecting right now.
AI penetration testing is a structured security assessment that identifies exploitable weaknesses in AI systems, including large language models, computer vision pipelines, and autonomous decision engines. Unlike traditional penetration testing, it addresses AI-specific attack surfaces such as prompt injection, model exfiltration, data poisoning, and jailbreaking.
Prompt injection is an attack where malicious instructions are embedded in content that an AI model processes. The model interprets those instructions as legitimate commands and follows them. It is classified as LLM01 in the OWASP Top 10 for Large Language Models and represents one of the most direct attack paths against AI systems that process user-controlled input.
We use PyRIT and garak to orchestrate prompt injection and jailbreak attacks at scale. For prompt mutation, FuzzLLM tests thousands of structural variations to find novel jailbreaks. For vision, audio, and tabular ML components, IBM’s Adversarial Robustness Toolbox generates perturbations, poison samples, and extraction queries tailored to each modality.
Automated testing provides breadth, running thousands of attack variants to identify potential weaknesses quickly. Manual testing provides depth. A skilled operator can take a single finding from automated testing and expand it into a multi-turn attack that exfiltrates system prompts, API keys, or confidential data. Chained attacks that cross multiple layers, such as storage, parsing, retrieval, and inference, require manual testing because no single automated tool exercises them end to end.
RAG stands for Retrieval-Augmented Generation. It is a technique where an AI model retrieves relevant content from a data store to inform its responses. Attackers can target RAG systems through indirect prompt injection, embedding malicious instructions in documents that get ingested into the vector store. When a user asks a normal question, the retriever surfaces the payload and the model executes the attacker’s instructions.
We map findings to the OWASP LLM Top 10, MITRE ATLAS, and the traditional MITRE ATT&CK framework. This allows security teams to slot each finding into a familiar defensive playbook and prioritize remediation based on their existing processes.
Clients receive a report with an executive narrative, a high-level analysis of tested components, and detailed vulnerability write-ups covering what was tested, what the model did, and why it matters. Each finding includes tailored remediation guidance. We also provide free remediation retesting once fixes are implemented and conduct interactive out-briefs with the engineering team.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.
