Welcome back for Challenge 2 of the AI Red-Teaming Playground Labs! If you missed part 1 (PyRIT Setup and Challenge 1) you’ll want to go back and read that. This challenge will be “metaprompt extraction using obfuscation and encoding techniques” and we’ll once again go through it manually and automated with PyRIT. We’ll also write some simple custom classes to demonstrate some of the PyRIT functionality.
Manual was simple enough again. Backwards worked:
Base64 also worked:
Let’s play around with some custom classes here to get more familiar with the functionality. First, there is a built-in Base64Converter class that tries to turn the whole conversation into base64. I’ll make a much simpler converter called Base64RequestConverter that simply adds “In base64, ” to the beginning of a given prompt, making it similar to my manual attempt. I’ll also try using a Scorer to automate success validation.
In my testing I also saw some instances where after revealing the flag, further attempts within the same chat instance would always reveal the flag (or vice-versa – successful prompts from previous chats failed after multiple rejected attempts within a new chat), so the chat context history may be influencing responses. To avoid this, I set it up to make an initial request to the New Chat endpoint (http://localhost:4002/chats) and grab the new chat ID in the response so every message went to a brand new chat. Understanding usage and context for the specific target under test will be key to more effectively red teaming it at scale.
Building a Converter
Copying the Base64Converter class and simplifying it is easy enough. I won’t be sharing any code here, but I recommend looking through the default converters’ code and creating something basic to get familiar with the structure and capabilities.
My new Base64RequestConverter worked as expected – it added “In base64, ” to the front of the prompt and extracted the secret word encoded in base64:
Setting up scorer
Looking through the built-in scorers, there’s one called SubStringScorer that is as straightforward as it gets – it just checks the response for a specified substring. For this challenge, that won’t work because the secret word may be base64-encoded in the response. We’re going to need a scorer that can decode responses if needed.
First let’s test the SubStringScorer to make sure things work as expected. I pulled up the notebook from Challenge 1, pointed it at this challenge, and added the scorer with the substring of the secret word that we now know. I added the secret word to the prompt to make sure the chatbot put it in the response.
At the bottom you can see that the SubStringScorer returned True, as the target substring was found in the response. Let’s make a new scorer called SubStringScorerWithBase64 that is a copy of SubStringScorer with an additional check for base64 content. Now we can catch successful plain text and encoded responses. Again, I leave you to create your own scorer, but it was fairly straightforward when starting with a default scorer class.
Using our new classes to automate testing
Now that we have a decoding scorer, we have the power to automate the full process. We know that base64 encoding is one way to get the answer, but the chatbot didn’t always just give it up right away – phrasing made a difference. Let’s run it through the Base64RequestConverter combined with the ToneConverter.
I added ten various tones to cycle through. The GPT-4o generated tone conversions could get entertaining – “mysterious” was especially fun to read:
And the scoring worked! Not only was it a time-saver while manually looking through output, but it also gives us the ability to add some automated analysis. Since order of Converters matters, I was curious which process would be better: Base64 then Tone (prepend “In base64, ” to the original prompt and then change tone), or Tone then Base64 (tone first then prepend “In base64, ” to the tone-converted prompt). I set up a loop to cycle through each tone variation 10 times per scenario, for a total of 100 requests each:
Look at all the insight we can draw from this quickly without having to manually review chat output. Overall, dramatic and mysterious tones were most successful, with urgent and academic requests close behind. On the flip side, sarcasm did not go over well with the chatbot. Tone change first was also noticeably better – digging into it deeper, the tone conversion sometimes garbled the base64 part (especially for the academic tone, where it often mistakenly asked it to “decode the base64” instead of encode) whereas the direct base64 intro left no room for confusion. This makes sense as the challenge was specifically designed to be more revealing to encoding attempts – my testing using Tone only resulted in 0% success rate without the Base64.
As a red teamer, this type of analysis is key to designing attack paths. When first starting to interact with a target LLM application, I can pick a basic goal of something the application shouldn’t respond with and run through a bunch of conversions to find what works and what doesn’t. Then I can craft more impactful attacks using that insight. Step 1 might be getting it to explain how to make a harmful substance; Step 2 would be using the most successful techniques validated through Step 1 to extract RAG documents, execute commands, poison organizational data, or carry out any other assessment goals.
Just like when socially engineering people, different AI applications will be susceptible to different manipulations. Utilizing PyRIT to automate and analyze bypass success can provide a major boost to abusing LLMs, and in turn provide organizations with more accurate risk analysis and reduction as they implement AI into their processes.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.
Software testing is more than just a step in the development process; it’s a critical practice that drives the quality, security, and reliability of the software we deliver. At BreakPoint Labs, we’ve integrated rigorous testing throughout the entire development lifecycle, ensuring that every piece of software we produce meets our high standards and aligns with the needs of our clients and end users’ expectations.
Integrated Testing Throughout the Lifecycle
We believe in embedding testing into every phase of the development lifecycle, starting from the earliest stages and continuing through to post-deployment. By designing and executing tests from the beginning, issues or bugs are caught early, preventing them from escalating into larger problems. This proactive approach, combined with continuous testing as the software evolves, ensures a smoother development process and results in a more robust, reliable final product or service.
Embracing Automation
Leveraging technology to deliver effective and sustainable capabilities is a core tenant of BreakPoint Labs. Automation is a key part of our testing strategy. By automating repetitive and complex tests, we can quickly verify that our software works as intended under various conditions. This not only saves time but also improves accuracy and consistency, giving us the confidence that our software and services can handle real-world demands under realistic constraints and against sophisticated actors.
A Comprehensive Testing Strategy
Our approach to testing is comprehensive and covers every aspect of the software:
Each of these testing methods is designed to address specific aspects of the software, ensuring that the final product is well-rounded and ready for deployment.
Building Confidence and Streamlining the ATO Process
Rigorous testing isn’t just about making sure the software works—it’s about building trust. Thorough testing gives stakeholders the confidence that the software will perform reliably in the real world.
Additionally, our detailed testing process supports the Authority to Operate (ATO) process. By documenting every step of our testing, we create artifacts that are essential for gate analysis and compliance reviews. These artifacts demonstrate that the software meets all security and operational requirements, making it easier to secure the necessary approvals and get the software into the hands of users quickly and efficiently. A good example that we will dig into is “chaos testing”.
Chaos Testing
When writing chaos tests, the objective is to introduce failures and observe how the system responds to ensure resilience. Using a tool like ChaosToolkit, you can simulate various failure scenarios, such as resource constraints or random service disruptions. In one of my tests on a sample Django application, we found that it handled the load when utilizing 1-3 CPU cores but failed when pushed to 4 cores, uncovering scalability issues. In this stress test, the tool was configured to fully utilize 4 CPU cores for 10 seconds, simulating a heavy computational workload that the application could not handle under the increased pressure. Another test involved randomly restarting containers to verify they would come back up and remain accessible, which is essential for maintaining high availability. When writing chaos tests, start with a clear hypothesis, such as ensuring services remain functional under stress or after restarts. Use tools like ChaosToolkit to automate these tests in controlled environments, and always monitor logs for unexpected behaviors.
Test Scenarios utilized ChaosToolKit, Django, and Docker
Django Application successfully handling 3 CPU cores of stress:
Django Application failing to handle 4 CPU cores of stress:
In summary, our software testing approach is more than just catching bugs. It’s about ensuring quality, building trust, and supporting compliance so that every piece of software we deliver is ready to perform when it matters most.
