During the initial phases of a penetration test, online resources such as Shodan can serve as a starting point to identifying the technology footprint of your target organization.
A new resource has been released, known as Censys.io, that collects data on internet facing websites and systems through daily ZMap and ZGrab scans of the IPv4 address space.
These online resources can be leveraged together or separately with various search parameters (such as TLD, CIDR, etc.) to query the available databases, helping to identify any information and services exposed to the Internet.
The four primary ways these resources are leveraged:
- Query a CIDR to enumerate systems and services (take note of any additional domains that are enumerated)
- Query a Top Level Domain (TLD) to enumerate systems and services (take note of any additional CIDR ranges that are enumerated)
- Query for a single target system
- Query for a search string (Example: Apache, etc.)
It is often useful to show a customer the external host and service exposure, and this process will commonly lead to the scope being expanded to include previously “unknown” hosts/services. Both Shodan and Censys.io offer a GUI to interact with the data, though both provide direct access to the data through an API.
Shodan:
Shodan is an excellent resource for external service enumeration, and supports queries of both IPv4 and IPv6 data. Shodan offers both a paid and free edition for API access and extended GUI access, as well as a Python module to make data extraction easier. Below is an example of interacting with Shodan’s free API access in Python:
[code language=”py”]
~$ python
>>>
>>> import shodan
>>> SHODAN_API_KEY = ‘[INSERT_API_KEY]’
>>> api = shodan.Shodan(SHODAN_API_KEY)
>>> results = api.host("8.8.8.8")
>>>
>>> results.keys()
[u’data’, u’city’, u’region_code’, u’ip’, u’isp’, u’area_code’, u’dma_code’, u’last_update’, u’country_code3′, u’latitude’, u’hostnames’, u’postal_code’, u’longitude’, u’country_code’, u’org’, u’country_name’, u’ip_str’, u’os’, u’asn’, u’ports’]
>>> results[‘ip_str’]
u’8.8.8.8′
>>> results[‘ports’]
[53]
>>> results[‘hostnames’]
[u’google-public-dns-a.google.com’]
>>>
[/code]
With the premium Shodan API access, you can implement search filters when performing queries, drastically increasing the functionality and level of detail you can use when searching for data. A more advanced example of Shodan scripting which requires paid API access is demonstrated below. The output has been abbreviated, but you can see this can give you a quick snapshot of external services, network ranges, TLDs, etc. that can be used to drive your reconnaissance and scoping process further.
[code language=”py”]
$ shodan_searcher -s hostname:google.com
[+] Query: hostname:google.com Total_Results: 44510
[-] 8.8.8.8 (google-public-dns-a.google.com) 53
[-] 178.35.137.68 (cache.google.com) 80
[-] 82.223.249.194 (www.google.com) 3306 Server: MySQL
[-] 208.117.228.231 (cache.google.com) 80 Server: Google httpd
[-] 80.228.66.209 (ewe-hb-GGC-Node3-209.cache.google.com) 80
[-] 179.49.8.116 (cache.google.com) 80
[-] 37.98.225.196 (cache.google.com) 80
…..
[+] Found Results in CIDRs: 208.117.244.0/24, 186.208.224.0/21, 85.118.96.0/19, 210.245.8.0/21, 86.120.0.0/13, 173.237.96.0/19, 114.31.0.0/20, 201.218.92.0/24, 213.153.32.0/19, 193.95.0.0/17, 177.155.141.0/24, 212.2.96.0/19, 212.113.48.0/20, 219.88.188.0/22, 83.174.192.0/18, 103.10.20.0/22, 202.163.106.0/24, 217.64.139.0/24,
…..
202.69.184.0/22, 192.232.16.0/20, 95.180.0.0/17, 189.28.163.0/24, 177.91.191.0/24, 206.80.240.0/20, 180.150.0.0/17, 217.175.200.0/24, 8.8.4.0/24, 176.126.56.0/22, 37.98.225.0/24, 77.120.60.0/22, 212.10.0.0/16, 109.110.32.0/23, 203.2.228.0/24, 77.50.0.0/17, 27.109.0.0/24, 95.181.0.0/17, 209.91.128.0/18
[+] Found the following TLDs: google.com
[/code]
**Recon-ng and Maltego also can tie into Shodan and offer increased functionality by configuring the tool with an API key.
Censys:
Censys.io is a new free online resource that was created by researchers at the University of Michigan. This project offers a web services API, as well as a Python module to improve API interaction. The Censys.io API can be utilized with a free account once created. In order to perform API queries, you will need the API ID and Secret key values associated with your account. Both of these unique key values can be found under the “My Account” section. Below is a quick example of interacting with the Censys.io API in Python:
[code language=”py”]
$ python
>>>
>>> import censys
>>> from censys import *
>>>
>>> api = censys.ipv4.CensysIPv4(api_id="[INSERT_KEY]", api_secret="[INSERT_KEY]")
>>> res = api.search("ip:8.8.8.8")
>>> res
{u’status’: u’ok’, u’results’: [{u’ip’: u’8.8.8.8′, u’protocols’: [u’53/dns’]}], u’metadata’: {u’count’: 1, u’query’: u’ip:8.8.8.8′, u’backend_time’: 34, u’page’: 1, u’pages’: 1}}
>>> for i in res.get(‘results’):
… print "{} {}".format(i.get("ip"), " ".join(i.get(‘protocols’))
…
8.8.8.8 53/dns
>>>
[/code]
Below is a POC script that extracts data from the API, by default only grabbing the first page of results (100 results). To circumvent this restriction, we added some simple logic to parse the total and make the appropriate number of queries.
[code language=”py”]
#!/usr/bin/env python
import sys
import censys
from censys import *
api = censys.ipv4.CensysIPv4(api_id="API_ID", api_secret="API_SECRET")
res = api.search(sys.argv[1])
matches = res[‘metadata’][‘count’]
pageNum = matches / 100
if matches % 100 != 0:
pageNum = pageNum + 1
count = 1
while count <= pageNum:
res = api.search(sys.argv[1], page=count)
count = count+1
for i in res.get(‘results’):
print "{} {}".format(i.get("ip"), " ".join(i.get(‘protocols’))
[/code]