Context discovery is a critical part of an in-depth application testing methodology. This is an area where loads of risk can be uncovered that web application scanners and past security assessments might have missed. At BreakPoint Labs (BPL), we place a strong focus on application content discovery. This blog post will share our approach to content discovery, with the hope of inspiring others to dig deeper in their next application security assessment.

If your content discovery workflow is just pointing ffuf or gobuster at a target with a giant generic wordlist, you’re probably leaving a massive amount of attack surface on the table. While there is nothing inherently wrong with these tools or using large generic wordlists, there are better ways.

Oftentimes, we get in a “one size fits all” mindset when it comes to specific workflows in our methodology. Run the same tools and syntax for each engagement, hoping to find everything in a single easy command. But think about it, if we are scanning a web application for unlinked content, do we think we are going to find all those hidden endpoints with one list? Doubtful.

A smarter approach to unlinked content is to build it in stages. Focus on the target. What tech stack is running on the application? What endpoints have we already discovered? What endpoints or words/patterns do we already know about? These are all questions we should ask ourselves when picking wordlists, and remember, this doesn’t have to be done in one large sweep. In fact, it is often better to do this in stages, analyzing what you have found (and not found) to better craft your next scan.

Lastly, consider the tools we are using for this. These content discovery tools are a dime a dozen. Ffuf, gobuster, dirsearch, feroxbuster, the list goes on. Nothing wrong with these tools, but they all basically do the same thing. So really, it’s up to you which one you like. If it works for you, use that. I typically stay within Burp Suite, using the Discover Content feature and sometimes Intruder. My main selling point here is simple: I can easily add what I discover to the site map, passively analyze the endpoints I discover for new linked content, and configure Burp Suite to leverage what it finds to build custom approaches for the specific target.

Leveraging the Power of Burp Suite Discover Content Tool

The tool can be accessed by right-clicking a host in your site map and going to Engagement Tools -> Discover Content. You will see three (3) tabs: Control, Config, and Site Map.

  • Control – Used to monitor the discovery scan to see tasks, errors, etc. Also allows you to start/stop the scan.
  • Config – Define how you want the scan to be conducted. More on this later
  • Site Map – Shows the site map of the current scan. Results will get added to the main site map unless specified otherwise in the Config tab.

Let’s dive into the Config tab, where we can really fine-tune the scans we run. While I won’t go into every possible configuration, I will cover each unique setting, what they do, and the specific use cases I like to use.

The Target section is where we can, of course choose the target, change the discover types (Files and/or Directories), and a recursion limit. I typically leave this as is, depending on the target, so not much to talk about here.

The Filenames section focuses on which wordlist(s) to use and how to configure them.

  • The Built-in options let Burp know if you want to use the build wordlists for content discovery. I rarely use these, but they could fit into your methodology if you choose to.
  • The Custom list options are where you would choose a wordlist you have on disk. We will go more into choosing wordlists later in this post.
  • The Names observed in use on target site checkboxes tell Burp to add the directory and filename stems it discovers to a list to check for each new directory it finds during its scan. This is a useful feature, but it will increase your scan time. I typically save this one for the latter stages of discovery.
  • Lastly, Derivations based on discovered items is where I really like using this tool over other fuzzers. When Burp discovers a new item (file or directory), it attempts to look for patterns and tests the application against those patterns. For example, per Burp Suite’s website, if Burp discovers the directory “AnnualReport2018”, it will also check for “AnnualReport2019”, “AnnualReport2020”, etc. I almost always leave this checked, as the overhead in most cases isn’t too bad, and I can potentially find new content that a static wordlist would never find.

The File extensions section is helpful for controlling the addition of various file extensions to file stems. When Burp tests for a file, it will also check for these additional file extensions configured in this section.

  • The Test these extensions tells Burp to always check for these extensions on files in your wordlist, regardless of whether the original file was found. I typically uncheck this box for most of my scans and enable it only for more tech-specific extensions that the next checkbox may not cover. We will cover this more in a later section of this post.
  • The Test all extensions observed…, except for tells Burp to test for all extensions observed on the target except for these. I typically leave this on all the time, depending on my specific use case. This is nice, since it will test for tech-specific extensions as it is using what has already been discovered on the site; however, it can miss stuff. This is where I will use the previous option to fine-tune based on the target site in an attempt to discover other content—more on that in a later section of this post.
  • Testing these variant extensions on discovered files is nice, for testing backup files and other variations of files that Burp has discovered. I typically leave this one enabled and may add a few variations depending on the target site. For example, if I see that the site is running ASP.NET, I will add variations such as “asp.bak”, “aspx.backup”, etc.
  • The Test file stems with no extension; just test the files in your wordlist(s) without an extension, which I typically just leave checked.

Lastly, the Discovery Engine section is used to control how the scan is used for things such as crawling the discovered content and interactions with the site map.

  • The Case sensitivity option lets you define if the target is case-sensitive when it comes to its content. I typically leave it as Auto-detect, but feel free to adjust based on your needs.
  • The Add discovered content to site map will automatically add discovered content to the main target site map. I always leave this on.
  • The Copy content from site map will add the content from the main targets site map to your discover site map, which could provide additional areas to discover content.
  • The Spider will automatically crawl any discovered pages.
  • Lastly, you can change the number of threads for the discovery and spider tool with the last 2 options.

So far, we have just talked about how we can configure this tool to dynamically add extensibility to our content discovery. Now let’s talk about authentication. Most tools, like ffuf, at least to my knowledge, don’t have a great way to handle this. You can pass in authentication headers and cookies into these tools, but when the session expires, there is no automated way to handle it.

This is another perk of using Burp Suite’s Discover Content tool: it can be used with Burp’s built-in session handling and cookie jar features to automate this, making authenticated discovery easier. We won’t go into how to set up session handling rules, macros, or the cookie jar, as there are plenty of resources available for this. However, when using these features with the Discover Content tool, be sure to update the Use cookies from Burp’s cookie jar rule to include both the Target and Scanner tools. This will ensure your scan uses the cookie(s) in place there.

Words are Hard: Choose the Right Wordlist

Now let’s talk about wordlists. What to use, when to use it, how to use it, and why it even matters. I see a lot of times people do one of two things: they either use a large, generic, publicly available wordlist that just hits on all cylinders. Or, they build custom wordlists that, well, are just collections of other wordlists that again hit on all cylinders. Don’t get me wrong, this, in and of itself, is not a bad idea. The problem is this is where they stop, but for me, this is just the beginning.

The Baseline

This is stage one; these are generic wordlists I like to use to see what’s out there. These are the classic wordlists we have all seen used. But they can get us a good idea of what is going on with the target and, in some cases, find easy low-hanging fruit. A few examples I like to use for this are:

Target Specific

Once you have completed a thorough walk-through of the application, maybe even crawled the site and conducted some unlinked content discovery using the lists above, it’s time to fine-tune your next phase.

I like to start by looking at the site map, noting all extensions I see, headers that can help identify the tech being used, interesting endpoints such as /api, /admin, /wp-content, etc. Also, don’t forget to look at things like Wapalyzer and passive collection notes in Burp to find additional tech you may not be tracking.

Once this data has been collected, it’s time to plan the next phase. Find wordlists that are specific to the application. For example, if the site is running WordPress, then use a WordPress-specific wordlist. If the site has a lot of php, then find wordlists targeting php endpoints, etc. This seems basic, but a lot of times people don’t take the time to really plan out this phase, and instead just try to throw everything at once and move on.

I’ll give a few examples I like to use here, but keep in mind there are plenty more out there. Do your research and find what you like.

Custom Wordlists

Last tip I will give here is, be sure to also use custom wordlists. Starting with public wordlists like this is a great place to start, but there can be real power in using custom wordlists, not only for specific targets using tools like GAP and backup-gen, but for your own collection of private wordlists that can be used for other engagements. When you find endpoints or filenames that look interesting for a site you are targeting, be sure to add them to your wordlists wherever you find the most valuable, such as php files to your php-specific wordlists. Never know when you may find it again.

Another valuable thing when you have a large scope of many applications on a target domain is to use words, parameters, etc, from the other sites on the site you are focusing on. These sites could share similar patterns depending on the developers. For example, if you have dev sites in your scope, you may find that they use various parameters that are used in dev but that have made it into production. So be sure to grab these with GAP and try them on your site.

Putting it All Together

Let’s run through a quick example, putting all this information together to show how I typically approach new applications to test.

  • STEP 1: Install passive tools like JS Link Finder, which can help find linked content that Burp parsers may miss in files such as JavaScript. This all runs in the background, so setting up first ensures we have the best chance of finding these endpoints while navigating the site. There are plenty of these out there; I just like JS Link Finder.
  • STEP 2: Walk the application, get a feel for what it is doing, how it passes data, what tech stack it uses, etc.
  • STEP 3: Crawl the site. Analyze this data in the site map for interesting patterns.
  • STEP 4: Baseline unlinked content discovery. Use the wordlists discussed above, or other lists you find, to perform a baseline of common endpoints.
  • STEP 5: Analyze what you have.
    • Look at your site map, see what has changed. Look for patterns and, if needed, repeat step 4 on directories you may have missed.Look at your passive collection output. Anything interesting in the JavaScript files? This could show a whole new list of endpoints you may have missed.
    • Look for tech-specific data, such as specific endpoints, metadata in the source code, the favicon hash, etc.
  • STEP 6: Use target-specific wordlists. This will likely be done in several stages, including various tech-specific wordlists based on your target, like PHP, Tomcat, IIS, etc. Also, don’t forget about tools like GAP to get custom wordlists to use for parameters, files, directories, etc.
  • STEP 7: Analyze what you have.
    • Repeat any of the above steps as needed, targeting different directories you may not have recursively hit.Extract interesting target endpoints to attack. APIs, login portals, forms you can submit data, interesting parameters in the body/url, etc
    • Speaking of parameters, don’t forget to look for unlinked parameters as well. Just as unlinked files and directories have been the main focus of this post, unlinked parameters can be vulnerable to many different attacks, depending on the target. Use tools like ParamMiner for this and don’t forget to try different wordlists, both public wordlists from places like SecLists and custom ones pulled from GAP

Summary

In summary, Burp Suite’s Discover Content tool, when configured properly, can greatly improve content discovery and, in my experience, find more content than other fuzzers. I’m not saying other tools are bad, nor that they don’t provide functionality that beats this tool. However, I wanted to take this opportunity to show the power of understanding your toolkit and leveraging all the features it has to offer to improve your workflow. If you already have a tool like this built into your methodology and don’t want to change, I highly encourage you to explore all the features your tool offers. Test out different configurations and wordlists, find what works best for you. In the end, the tool you use is subjective, but I hope this article helps you improve how you approach content discovery.

Author Image

About the Author

Trevor Isenberg is a Senior Red Team Operator at BreakPoint Labs for a DoD-certified Red Team, responsible for assessing and securing critical ICS/SCADA assets. Coming from a background in system administration, Trevor provides a unique perspective when targeting complex networks and solving challenges. He holds numerous professional certifications and is especially interested in Web Application, Active Directory and ICS/SCADA pentesting. When he’s not in the middle of a red team engagment, you’ll find him spending time with family, playing guitar and working on projects at night to continuing growing his skillset.