Context discovery is a critical part of an in-depth application testing methodology. This is an area where loads of risk can be uncovered that web application scanners and past security assessments might have missed. At BreakPoint Labs (BPL), we place a strong focus on application content discovery. This blog post will share our approach to content discovery, with the hope of inspiring others to dig deeper in their next application security assessment.
If your content discovery workflow is just pointing ffuf or gobuster at a target with a giant generic wordlist, you’re probably leaving a massive amount of attack surface on the table. While there is nothing inherently wrong with these tools or using large generic wordlists, there are better ways.
Oftentimes, we get in a “one size fits all” mindset when it comes to specific workflows in our methodology. Run the same tools and syntax for each engagement, hoping to find everything in a single easy command. But think about it, if we are scanning a web application for unlinked content, do we think we are going to find all those hidden endpoints with one list? Doubtful.
A smarter approach to unlinked content is to build it in stages. Focus on the target. What tech stack is running on the application? What endpoints have we already discovered? What endpoints or words/patterns do we already know about? These are all questions we should ask ourselves when picking wordlists, and remember, this doesn’t have to be done in one large sweep. In fact, it is often better to do this in stages, analyzing what you have found (and not found) to better craft your next scan.
Lastly, consider the tools we are using for this. These content discovery tools are a dime a dozen. Ffuf, gobuster, dirsearch, feroxbuster, the list goes on. Nothing wrong with these tools, but they all basically do the same thing. So really, it’s up to you which one you like. If it works for you, use that. I typically stay within Burp Suite, using the Discover Content feature and sometimes Intruder. My main selling point here is simple: I can easily add what I discover to the site map, passively analyze the endpoints I discover for new linked content, and configure Burp Suite to leverage what it finds to build custom approaches for the specific target.
The tool can be accessed by right-clicking a host in your site map and going to Engagement Tools -> Discover Content. You will see three (3) tabs: Control, Config, and Site Map.
Let’s dive into the Config tab, where we can really fine-tune the scans we run. While I won’t go into every possible configuration, I will cover each unique setting, what they do, and the specific use cases I like to use.
The Target section is where we can, of course choose the target, change the discover types (Files and/or Directories), and a recursion limit. I typically leave this as is, depending on the target, so not much to talk about here.
The Filenames section focuses on which wordlist(s) to use and how to configure them.
The File extensions section is helpful for controlling the addition of various file extensions to file stems. When Burp tests for a file, it will also check for these additional file extensions configured in this section.
Lastly, the Discovery Engine section is used to control how the scan is used for things such as crawling the discovered content and interactions with the site map.
So far, we have just talked about how we can configure this tool to dynamically add extensibility to our content discovery. Now let’s talk about authentication. Most tools, like ffuf, at least to my knowledge, don’t have a great way to handle this. You can pass in authentication headers and cookies into these tools, but when the session expires, there is no automated way to handle it.
This is another perk of using Burp Suite’s Discover Content tool: it can be used with Burp’s built-in session handling and cookie jar features to automate this, making authenticated discovery easier. We won’t go into how to set up session handling rules, macros, or the cookie jar, as there are plenty of resources available for this. However, when using these features with the Discover Content tool, be sure to update the Use cookies from Burp’s cookie jar rule to include both the Target and Scanner tools. This will ensure your scan uses the cookie(s) in place there.
Now let’s talk about wordlists. What to use, when to use it, how to use it, and why it even matters. I see a lot of times people do one of two things: they either use a large, generic, publicly available wordlist that just hits on all cylinders. Or, they build custom wordlists that, well, are just collections of other wordlists that again hit on all cylinders. Don’t get me wrong, this, in and of itself, is not a bad idea. The problem is this is where they stop, but for me, this is just the beginning.
This is stage one; these are generic wordlists I like to use to see what’s out there. These are the classic wordlists we have all seen used. But they can get us a good idea of what is going on with the target and, in some cases, find easy low-hanging fruit. A few examples I like to use for this are:
Once you have completed a thorough walk-through of the application, maybe even crawled the site and conducted some unlinked content discovery using the lists above, it’s time to fine-tune your next phase.
I like to start by looking at the site map, noting all extensions I see, headers that can help identify the tech being used, interesting endpoints such as /api, /admin, /wp-content, etc. Also, don’t forget to look at things like Wapalyzer and passive collection notes in Burp to find additional tech you may not be tracking.
Once this data has been collected, it’s time to plan the next phase. Find wordlists that are specific to the application. For example, if the site is running WordPress, then use a WordPress-specific wordlist. If the site has a lot of php, then find wordlists targeting php endpoints, etc. This seems basic, but a lot of times people don’t take the time to really plan out this phase, and instead just try to throw everything at once and move on.
I’ll give a few examples I like to use here, but keep in mind there are plenty more out there. Do your research and find what you like.
Last tip I will give here is, be sure to also use custom wordlists. Starting with public wordlists like this is a great place to start, but there can be real power in using custom wordlists, not only for specific targets using tools like GAP and backup-gen, but for your own collection of private wordlists that can be used for other engagements. When you find endpoints or filenames that look interesting for a site you are targeting, be sure to add them to your wordlists wherever you find the most valuable, such as php files to your php-specific wordlists. Never know when you may find it again.
Another valuable thing when you have a large scope of many applications on a target domain is to use words, parameters, etc, from the other sites on the site you are focusing on. These sites could share similar patterns depending on the developers. For example, if you have dev sites in your scope, you may find that they use various parameters that are used in dev but that have made it into production. So be sure to grab these with GAP and try them on your site.
Let’s run through a quick example, putting all this information together to show how I typically approach new applications to test.
In summary, Burp Suite’s Discover Content tool, when configured properly, can greatly improve content discovery and, in my experience, find more content than other fuzzers. I’m not saying other tools are bad, nor that they don’t provide functionality that beats this tool. However, I wanted to take this opportunity to show the power of understanding your toolkit and leveraging all the features it has to offer to improve your workflow. If you already have a tool like this built into your methodology and don’t want to change, I highly encourage you to explore all the features your tool offers. Test out different configurations and wordlists, find what works best for you. In the end, the tool you use is subjective, but I hope this article helps you improve how you approach content discovery.
Trevor Isenberg is a Senior Red Team Operator at BreakPoint Labs for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. Coming from a background in system administration, Trevor provides a unique perspective when targeting complex networks and solving challenges. He holds numerous professional certifications and is especially interested in Web Application, Active Directory and ICS/SCADA pentesting. When he’s not in the middle of a red team engagment, you’ll find him spending time with family, playing guitar and working on projects at night to continuing growing his skillset.
