Thanks for joining me for the second installment on leveraging Trackers in Microsoft Defender External Attack Surface Management (Defender EASM) to find and manage risk in your organization. This blog post is part two of this series, building on the concepts introduced in part one about discovering your attack surface and applying this valuable inventory data to inform your security efforts at scale. As a quick refresher, in part one, we defined Trackers in Defender EASM and learned how to search for them in the User Interface (UI). This blog post will closely examine the Defender EASM Application Program Interface (API).
Concepts: The Defender EASM API
The Defender EASM API supplies a much more detailed view of an organization’s attack surface and allows end users to automate processes and operationalize workflows using standardized REST API calls. In this blog post, we will walk through a simple example of extracting “attributetypes” and “attributeValues” from the JSON (JavaScript Object Notation) responses returned by the Defender EASM API. We should remember from part one of this series that in Defender EASM, the terms “Tracker” and “Attribute” are synonymous. This asset metadata has entries for the type of tracker/attribute and a value for the tracker/attribute.
Let us begin with using the API to answer a real-world question. Defender EASM power users with a keen eye may already know that the UI does allow end-users to search for specific attributes and values. But what if you do not know what attributes to search for? The Defender EASEM UI does not supply a mechanism to list all available attributes in an attack surface, but the API can help us see what is present within our own set of assets. We will use the API to answer the question, “What attributes (i.e., trackers) did Defender EASM find in my attack surface, how many of each type are there, and what are the attribute values?” Easy enough, but we must get some prerequisites out of the way.
Prerequisites
- Defender EASM resource deployed and populated with assets
- A cursory understanding of how REST APIs work and familiarity with interacting with them in your programming language or tool of choice
- Knowledge of the Defender EASEM API endpoints, specifically the “List” endpoint
- A Client Service Principal configured with the correct roles for access to the Defender EASM API
- A willingness to dive in, have fun, and get your hands dirty!
Concepts: The Approach
We will be modifying a Defender EASM example Jupyter Notebook published to GitHub. To succeed in this exercise, you should read the accompanying blog “Seeking Dead and Dying Servers with the MDEASM APIs” for more information and setup requirements. Our simple approach for retrieving all “attributeTypes,” “attributeValues,” and asset “names” are:
- Get a token for your Client Service Principal.
- Set a filter for all assets that are: “state = confirmed AND kind = page AND attributeType !empty.”
- Send an initial API request with the parameter “mark = *”
- Loop through subsequent API requests and JSON result sets to parse recent asset “name, “attributeType,” “attributeValue”, and add these to a dictionary.
- For each successful request, grab “next link,” resubmit, and append parsed results to the dictionary.
- Format and print results.
The example Jupyter Notebook will perform most of these steps for us, and we’ll simply need to focus on our query and desired output. To make our output a little easier to read, we will modify our Jupyter Notebook to use the pandas Python package.
Python Example
Assuming that we have followed all of the instructions to install the example Jupyter Notebook, we must ensure that we have the pandas package installed in our Python environment (virtual environments are recommended). Then, we will need to import pandas into our notebook by adding one line:
|
import requests, time, json, re # Add pandas package import here import pandas as pd |
For simplicity’s sake, we will also make a copy and modify an existing function in our Jupyter notebook, specifically, the function “get_asset_list().” In the Jupyter notebook, create a new markdown and a code cell just after the cell containing our imports, and paste the following code in each:
| ### Attribute Assets - List |
|
## Attribute Assets List returns## def get_attributeType_list(query): ''' Call Assets List endpoint and return Attributes - takes a URL encoded query string ''' check_url_encoding(query) global planeType planeType = 'data' azure_auth() nextUrl = None url = f"https://{region}.easm.defender.microsoft.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/workspaces/{resourceName}/assets?api-version={apiVersion}&maxpagesize={maxpagesize}&filter={query}&mark={mark}" # Code Modification: Create dictionary results = [] payload={} headers = { ‘User-Agent’: ‘MDEASM Python Notebook,’ ‘Authorization’: f’Bearer {bearerToken}’ }
response = requests.request("GET", url, headers=headers, data=payload) if response.status_code != 200: print("Error getting asset list") print(response.text) return None if len(response.text) == 0: print("No assets found matching your query") return None else: responsejson = json.loads(response.text) responseresults = responsejson['content'] # Code Modification: Loop, extract, and append attribute data points of interest for name in responseresults: for attributes in name['asset']['attributes']: if (attributes['recent']): r = { 'name' : name['name'], 'attributeType' : attributes['attributeType'], 'attributeValue' : attributes['attributeValue'], } results.append(r) if 'nextLink' in responsejson and nextUrl != responsejson['nextLink']: nextUrl = responsejson['nextLink'] else: nextUrl = None
while nextUrl: azure_auth() response = requests.request("GET", nextUrl, headers=headers, data=payload) responsejson = json.loads(response.text) responseresults = responsejson['content'] # Code Modification: Loop, extract, and append attribute data points of interest for name in responseresults: for attributes in name['asset']['attributes']: # Ensure only recent attribute data is returned if (attributes['recent']): r = { 'name' : name['name'], 'attributeType' : attributes['attributeType'], 'attributeValue' : attributes['attributeValue'], } results.append(r) if 'nextLink' not in responsejson: nextUrl = None else: time.sleep(1) nextUrl = responsejson['nextLink']
if len(results) == 0: print("No assets found matching your query") else: print("Number of results: " + str(len(results))) # Code Modification: return results instead of printing them return results |
Viewing Trackers with Pandas
Now that we’ve added a new function to our Jupyter Notebook, it’s time to use it and visualize the data with pandas. We’ll author a new query that only returns page assets that have attributes (i.e., trackers) and populate a pandas data frame with the following code in a new cell.
|
# Create a query to return page assets with non-empty attributes query = 'state = confirmed AND kind = page AND attributeType !empty' # Load our pandas dataframe df = pd.DataFrame(get_attributeType_list(query)) |
Once our query completes, Jupyter will print the results in a table with the following:
|
# View results returned df |
View the Most Common Trackers in an Attack Surface
Now, let’s use pandas to view just the counts of unique “attributeTypes." This list is useful for understanding which technologies are most prevalent in our attack surface and can be used as a starting point to inform and create more queries for hunting the data in your attack surface via the API. It’s also possible that you’ll see something unexpected in the list that requires more investigation to understand potential security risks.
|
# Create a second dataframe to display count of attributeTypes df2 = pd.DataFrame(df['attributeType'].value_counts().rename_axis('Attribute Type').reset_index(name='Count')) df2 |
Export Trackers Data? Yes!
Lastly, we’ll quickly export both of our data sets to *.csv files (comma-separated values) which can be viewed in Excel natively. Be sure to change the file path to match where the output file should be created on your computing device.
|
# Export our pandas dataframes to CSV df.to_csv (r'./export_all_attribute_data.csv', encoding='utf-8', index = None, header=True) df2.to_csv (r'./export_attributeType_count.csv', encoding='utf-8', index = None, header=True) |
We’ve barely scratched the surface of what the pandas Python module can do. I encourage everyone to learn more about its data analysis capabilities as this was a simple introduction. Python and pandas make it easy to quickly prototype visualizations, export data in standard formats, and help you quickly gain an understanding of the metadata in your own attack surface with the Defender EASM API.
Conclusion
Now you should have a good understanding of how to query for Trackers using the Defender EASM API. More importantly, I’ve shown you how to modify an example Jupyter Notebook published by Microsoft to create your own bespoke queries and visualizations. I hope you will join me for the next installment of this series, where we will look at a related asset metadata type – web components!
Be sure to try it for yourself! You can discover your attack surface discovery journey today for free.