Welcome back for Challenge 2 of the AI Red-Teaming Playground Labs! If you missed part 1 (PyRIT Setup and Challenge 1) you’ll want to go back and read that. This challenge will be “metaprompt extraction using obfuscation and encoding techniques” and we’ll once again go through it manually and automated with PyRIT. We’ll also write some simple custom classes to demonstrate some of the PyRIT functionality.
Manual was simple enough again. Backwards worked:
Base64 also worked:
Let’s play around with some custom classes here to get more familiar with the functionality. First, there is a built-in Base64Converter class that tries to turn the whole conversation into base64. I’ll make a much simpler converter called Base64RequestConverter that simply adds “In base64, ” to the beginning of a given prompt, making it similar to my manual attempt. I’ll also try using a Scorer to automate success validation.
In my testing I also saw some instances where after revealing the flag, further attempts within the same chat instance would always reveal the flag (or vice-versa – successful prompts from previous chats failed after multiple rejected attempts within a new chat), so the chat context history may be influencing responses. To avoid this, I set it up to make an initial request to the New Chat endpoint (http://localhost:4002/chats) and grab the new chat ID in the response so every message went to a brand new chat. Understanding usage and context for the specific target under test will be key to more effectively red teaming it at scale.
Building a Converter
Copying the Base64Converter class and simplifying it is easy enough. I won’t be sharing any code here, but I recommend looking through the default converters’ code and creating something basic to get familiar with the structure and capabilities.
My new Base64RequestConverter worked as expected – it added “In base64, ” to the front of the prompt and extracted the secret word encoded in base64:
Setting up scorer
Looking through the built-in scorers, there’s one called SubStringScorer that is as straightforward as it gets – it just checks the response for a specified substring. For this challenge, that won’t work because the secret word may be base64-encoded in the response. We’re going to need a scorer that can decode responses if needed.
First let’s test the SubStringScorer to make sure things work as expected. I pulled up the notebook from Challenge 1, pointed it at this challenge, and added the scorer with the substring of the secret word that we now know. I added the secret word to the prompt to make sure the chatbot put it in the response.
At the bottom you can see that the SubStringScorer returned True, as the target substring was found in the response. Let’s make a new scorer called SubStringScorerWithBase64 that is a copy of SubStringScorer with an additional check for base64 content. Now we can catch successful plain text and encoded responses. Again, I leave you to create your own scorer, but it was fairly straightforward when starting with a default scorer class.
Using our new classes to automate testing
Now that we have a decoding scorer, we have the power to automate the full process. We know that base64 encoding is one way to get the answer, but the chatbot didn’t always just give it up right away – phrasing made a difference. Let’s run it through the Base64RequestConverter combined with the ToneConverter.
I added ten various tones to cycle through. The GPT-4o generated tone conversions could get entertaining – “mysterious” was especially fun to read:
And the scoring worked! Not only was it a time-saver while manually looking through output, but it also gives us the ability to add some automated analysis. Since order of Converters matters, I was curious which process would be better: Base64 then Tone (prepend “In base64, ” to the original prompt and then change tone), or Tone then Base64 (tone first then prepend “In base64, ” to the tone-converted prompt). I set up a loop to cycle through each tone variation 10 times per scenario, for a total of 100 requests each:
Look at all the insight we can draw from this quickly without having to manually review chat output. Overall, dramatic and mysterious tones were most successful, with urgent and academic requests close behind. On the flip side, sarcasm did not go over well with the chatbot. Tone change first was also noticeably better – digging into it deeper, the tone conversion sometimes garbled the base64 part (especially for the academic tone, where it often mistakenly asked it to “decode the base64” instead of encode) whereas the direct base64 intro left no room for confusion. This makes sense as the challenge was specifically designed to be more revealing to encoding attempts – my testing using Tone only resulted in 0% success rate without the Base64.
As a red teamer, this type of analysis is key to designing attack paths. When first starting to interact with a target LLM application, I can pick a basic goal of something the application shouldn’t respond with and run through a bunch of conversions to find what works and what doesn’t. Then I can craft more impactful attacks using that insight. Step 1 might be getting it to explain how to make a harmful substance; Step 2 would be using the most successful techniques validated through Step 1 to extract RAG documents, execute commands, poison organizational data, or carry out any other assessment goals.
Just like when socially engineering people, different AI applications will be susceptible to different manipulations. Utilizing PyRIT to automate and analyze bypass success can provide a major boost to abusing LLMs, and in turn provide organizations with more accurate risk analysis and reduction as they implement AI into their processes.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.
Red teaming of AI-enabled systems is becoming essential for every organization to ensure they aren’t introducing risk to themselves and their stakeholders as they implement generative AI applications. Pentesters and red teamers need to become familiar with AI-specific attack techniques and terminology.
Microsoft has put out some AI Red-Teaming Playground Labs that are a great way to practice attacking AI applications. In this series of blog posts, we’ll walk through some of these challenges and uncover AI red-teaming tactics, techniques, and procedures (TTPs) that you can add to your arsenal.
The setup is a bit of effort initially, but I promise it’s worth it and won’t take much once it’s running. We’ll create an Azure account if you don’t already have one, set up an AI deployment for the Playground to use, then install and connect the Playground itself. Finally, effective offensive security testing should combine both automated and manual testing, so we’ll set up Microsoft’s PyRIT framework so we can walk through each challenge both manually and programmatically.
If you don’t have an Azure account, let’s set one up real quick. Cloud stuff always feels intentionally obscure and overly-complicated, so I’ll try to be detailed, but don’t blame me if this part is a pain. Also be aware of costs – new Azure accounts get $200 in free credits, and for this setup and Challenge 1 testing I’ve only used $0.09 of that, so no big deal here but just know that billing is happening in the background.
Here are the steps I took:
If all that worked successfully, congrats because it took me an embarrassingly long time to get to this point on the first pass. Let’s continue:
Success! Now you’ve got two models for the AI Red-Teaming Playground Labs to communicate with.
Install docker compose if you don’t have it already – if you’re using Kali like me, remember do not use apt install docker-compose! Do this instead:
apt install -y docker.io
mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose`
chmod +x ~/.docker/cli-plugins/docker-compose
docker compose versionThis is a great time to mention that docker is my archnemesis and it always causes me personal pain to recommend using it, but here we are.
Next is setting up the labs locally. Pull down the git repo:
git clone https://github.com/microsoft/AI-Red-Teaming-Playground-Labs.gitThe repo has a .env.example file in it. Copy this to .env and follow the instructions inside it to create a SECRET_KEY and an AUTH_KEY, then populate the rest with your Azure endpoint and API key saved previously. Set the model as “gpt-4o”.
And that should be it! Within that directory, run:
docker compose upThen navigate to http://localhost:5000/login?auth=<AUTH_KEY> and you should see a list of challenges to launch.
Finally, we’ll install PyRIT, which is Microsoft’s open-source framework for automating security testing of generative AI applications. Its documentation can also be obscure and needs a little more detail in my opinion, but we’ll do our best to figure it out.
Install pyrit, jupyter, and ipykernel:
pip install pyrit jupyter ipykernelInstall kernelspec:
python -m ipykernel install --user --name=pyrit_kernelSetup complete!
Challenge 1 – Direct Prompt Injection is straightforward and simple enough to do manually. The exact same prompt won’t work 100% of the time due to the nature of GenAI being non-deterministic and all that, but you shouldn’t have much of a problem convincing the chat bot to give up its passwords.txt.
I recommend poking around the PyRIT documentation and watching at least this high-level walkthrough from Microsoft. Even better, watch all 10 quick episodes of their AI Red-Teaming 101 course.
At a high-level, here’s what’s happening in PyRIT:
Datasets = Initial prompts to be fed into the pipeline.
Orchestrator = from their docs: “The Orchestrator is a top-level component that red team operators will interact with the most. It is responsible for telling PyRIT which endpoints to connect to and how to send prompts. It can be thought of as the component that executes an attack technique.”
Converters = converts the prompts into something else – it could be different encoding, putting prompts into a Word doc instead of just plain text, or much more complex things like the MathPromptConverter which is used to “transform user queries into symbolic mathematical problems by applying set theory, abstract algebra, and symbolic logic”. We’ll look at a few of these in a second.
Target = an AI target and how to connect to it. For this it will be an HTTPTarget but other examples include OpenAIChatTarget, AzureBlobStorageTarget, and HuggingFaceChatTarget.
ScoringEngine = to avoid manually reviewing every chat response, you can set up scorers that will automatically grade responses to see if they meet set criteria. We won’t do this today but you’ll want this when doing high-volume testing.
Ok let’s launch the Jupyter notebook. For some of these labs, Microsoft has included a notebook to start with for PyRIT testing.
jupyter notebookWhen the web GUI launches, select File -> New -> Notebook and choose the pyrit_kernel when prompted for Kernel. On the right, Upload and choose “AI-Red-Teaming-Playground-Labs/notebooks/Lab 1 – Credential exfiltraiton.ipynb”.
Follow the instructions to fill in the raw_http_request variable – don’t forget to adjust the curly braces according to step 7, and then put your prompt variable into the “input” parameter as shown below. Run the first cell and double check the print output looks as expected.
Note: if you ever get a JSONDecode error running the second cell (where it makes requests to the Lab 1 challenge HTTP endpoint) it’s likely due to a 401 response because the session cookie expired – you’ll need a new cookie/request here.
Note 2: if you see a duckdb error about unhashable type, this is a known bug in the newest duckdb (1.40), revert to 1.3.2 with pip install "duckdb==1.3.2"
Now that we’ve got a valid POST request that will send prompts to the Challenge 1 endpoint, let’s look into Converters to see how the initial prompt can be manipulated. Since I want to test out a bunch of Converters, in the first cell I changed the prompt converter import statement to get all of them:
from pyrit.prompt_converter import *In the second cell, when it defines the HTTPTarget, add the parameter “use_tls=False” at the end or else you’ll get SSL errors (as the challenges are HTTP only). It should now look like:
http_prompt_target = HTTPTarget(http_request=raw_http_request, callback_function=parsing_function, timeout=20.0, use_tls=False)CharacterSpaceConverter is a simple option to test first. It does exactly as advertised – puts a space between every character. Within the orchestrator, set the prompt_converter to CharacterSpaceConverter and run the notebook – you should see the converted prompt and the chat bot’s response.
Within the repo, you can find a list of all the built-in converters inside “PyRIT/pyrit/prompt_converter/__init__.py” (at the time of testing, I counted 61 total). Play around with different converters to see what options you have. You can stack multiple Converters as well and they’ll execute sequentially – here’s Leetspeak followed by CharacterSpace:
For a more complicated Converter, I tried MathPromptConverter, which frames the prompt as advanced math logic to see if it will complete the “math problem” which might get you past security controls. This and some other Converters require an LLM resource that PyRIT can use to generate advanced prompt mutations (and yes, you’re using GenAI to generate attacks against GenAI, this is life now).
Fortunately we just set up an Azure OpenAI endpoint for the Playground Labs and can re-use that same resource. You will need to define the endpoint details (found in Azure AI Foundry -> Deployment -> gpt-4o) within the code or in a .env file as explained in the PyRIT documentation.
We’ll do it directly in the code for now. Add an OpenAIChatTarget for this out-of-band prompt generation and give it the model name, API key, and endpoint. Within the orchestrator function call, add the MathPromptConverter and point it to the OpenAIChatTarget you just defined. The code should look similar to below (I’ve redacted my endpoint details):
Take a second to thank the universe you didn’t have to dig up old high school calculus textbooks to craft this prompt. Unfortunately here, the Challenge 1 chat bot didn’t fall for the math trap.
Playing with the Converters was interesting and there are plenty of techniques to try. The first to succeed in retrieving passwords.txt was the PersuasionConverter using the option “expert_endorsement”:
And that’s it for Challenge 1! Despite the simple example, it’s easy to see the potential here for automating large scale testing of AI implementations. We’ll be back soon for Challenge 2 – Metaprompt Extraction.
Ray Blasko is the Technical Director for offensive operations at BreakPoint Labs and a Red Team Operator for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. He is recognized as a subject matter expert in both attacking and defending IT and OT environments, and he excels at resolving the technical and strategic issues that arise in protecting critical infrastructure. Ray holds numerous professional certifications and regularly presents at DoD Red Team conferences, trade conventions, and information security events.