Blog Post

Microsoft Sentinel Blog
6 MIN READ

What’s new: Detect credential leaks using built-in Azure Sentinel notebooks!

Chi_Nguyen's avatar
Chi_Nguyen
Icon for Microsoft rankMicrosoft
Jun 01, 2021

Special thanks to ZhipengZhao , JulianGonzalezLars_Mohr, and Microsoft CredScanCore team for making these notebooks happen.

Thanks to Tiander Turpijn and Batami Gold for reviewing this blog and for their valuable feedback.

 

In this blog post, I’m going to walk you through three cool and easy-to-use Azure Sentinel notebooks that can scan logs across your entire Azure Sentinel workspace, Azure Blob storage, and Azure Data Explorer environment to detect credential leaks (which can save you from some serious potential cyberattacks!). These are the built-in templates that you can instantly use without writing any line of code!

 

Why should you care?

According to Verizon's 2020 Data Breach Investigation Report, the use of credentials in cyberattacks has been on a meteoric rise. Over 80% of hacking-related breaches involve the use of stolen or lost credentials.

 

It’s common sense to protect sensitive data such as passwords, API keys, database credentials, etc. by securely storing them. Unfortunately, storing data safely is not an easy task, and human error will continue to happen. This makes credential leaks high risks to many organizations. For that reason, it’s crucial to perform regular log scans to catch potential leaked credentials and take actions before they get in the wrong hands.

 

In the Azure Sentinel context, collected log messages are stored in a Log Analytics workspace. Many organizations also store their data in Azure Blob Storage or Azure Data Explorer, especially for long-term retention purpose. You might have an Azure Storage account Shared Access Signature used in a KQL query or an Azure Active Directory client access token used to authorize an application that has been hard-coded, logged and saved in a storage location. The storage becomes a gold mine for bad actors waiting to readily access, excavate, and exploit your organizations’ assets.

 

To help solve this problem, we’ve recently released three new Azure Sentinel notebooks that can scan across these environments – your Azure Sentinel workspace, Azure Blob Storage, and Azure Data Explorer - to uncover credential leaks in your data!

 

How do the notebooks work?

Each notebook scans logs in each respective environment.

  • The Credential Scan on Azure Log Analytics notebook enables you to pick any Azure Sentinel log table in your Log Analytics workspace and scan all or one specific column in the selected table.
  • The Credential Scan on Azure Data Explorer (ADX) notebook enables you to pick and scan a table in a database from a specific ADX cluster.
  • The Credential Scan on Azure Blob Storage notebook enables you to pick and scan a file in a blob container from your Blob storage account.

If any sensitive credentials are found, the results will be exported into a csv file. This file is saved in the same location as your notebook, where you can access and view the details of the leaked credentials. The file can also be downloaded and shared with relevant members in your team to validate the findings and apply appropriate remediation actions.

 

If no leaked credentials are found, no csv file is generated.

 

What types of credentials can the notebooks detect? The notebooks use regular expression patterns to identify the most common types of credentials, including passwords, Azure SQL connection strings, etc.  

For a full list of credential categories, please click on the following Spoiler section. 

Spoiler
  1. User login credentials
  2. Azure SQL connection string
  3. Azure IoT Shared Access Key
  4. Azure Storage Account Shared Access Signature
  5. Azure Storage Account Shared Access Signature for High-Risk Resources
  6. Azure Active Directory access token
  7. Amazon S3 client secret access key
  8. Azure Service Bus Shared Access Signature
  9. Azure Redis Cache Connection String Password
  10. Azure COSMOS DB Account Access Key
  11. Azure App Service Deployment Password
  12. Azure DevOps Personal Access Token
  13. Azure Function Master-API Key
  14. Azure Shared Access Key
  15. Azure AD Client Access Token
  16. X.509 Certificate Private Key
  17. ASP.NET Machine Key
  18. General Password
  19. Http Authorization Header
  20. Client Secret – API Key
  21. General Symmetric Key
  22. Ansible Vault
  23. Moniker Agent Storage Account Key
  24. Legacy Geneva Resource Access Key
  25. Domain User Password

Let’s look at a notebook example in detail. While these notebooks are simple to use, there are some basic pre-requisites and important instructions for them to work.

 

Pre-requisites

  • Data in at least one of these locations: Azure Sentinel workspace, Azure Blob storage, Azure Data Explorer.
  • An Azure Sentinel workspace to use Azure Sentinel notebooks.
  • An Azure Machine Learning (AML) workspace. Create an Azure Machine Learning workspace. You can set a default AML workspace from Azure Sentinel portal if you have more than one AML workspaces. Make sure to have at least Contributor permissions to the AML workspace to launch and execute the notebooks.
  • At least a Azure Sentinel or Azure Data Explorer or Azure Blob Storage Reader role to query logs on these data locations.

 

Deployment

  1. From the Azure Sentinel portal, navigate to the Threat Management section and open the Notebooks blade.
  2. Go to the Templates tab.
  3. Search for Credential Scan and you should see three notebooks in the result.
  4. Select one notebook. For this example, I’m going to use Credential Scan on Azure Data Explorer.
  5. On the right panel, select Save notebook. You can rename the selected notebook or keep the default name and save it to a default AML workspace. Then select OK.
  6. The notebook is now accessible to your AML workspace. From the same panel, select Launch notebook. Then you are prompted to log into the AML workspace.
  7. In the AML workspace, notice that a Credential Scan on Azure Data Explorer.ipynb file and a config.json file have been automatically generated from step 6 above.
    • The Credential Scan on Azure Data Explorer.ipynb file has the main content of the notebook.
    • The config.json file has configuration information about your Azure Sentinel environment where your notebook was launched from. It contains tenant_id, subscription_id, resource_group, workspace_id, and workspace_name, which are used for Azure Authentication (see step 9 below).
  8. Select a compute instance for your notebook server. If you don’t have a compute instance, create one by following step 5 in Launch a notebook using your Azure ML workspace.
 

create compute

 

  1. There are three main sections in this specific notebook: Warm-up, Azure Authentication, and Azure Data Explorer Queries. Each notebook cell contains instructions and/or the sample code using Azure SDK for Python and KQL. To avoid common errors, it’s important to not skip these instructions.
    • The Warm-up cells load Python libraries, parameters, and functions that will be used in the notebook.
    • The Azure Authentication section allows the notebook access to your Azure Resource Group where your ADX environment is located.
    • The Azure Data Explorer Queries section enables you to choose an ADX cluster, a database from the cluster, then a table that you want to scan.
  2.  After you finish running the notebook, if you don’t see the results in a newly created csv file, refresh the notebooks file explorer on the left navigation panel. Note that the name of the csv file is different for each notebook, and no csv file is created if no credentials are found.

csv file

 

11. If you need to download the csv file, select the three dots next to the file name, right click and hit Download option.

12. All scripts and output files are stored in a default storage account of your AML workspace.

  1. Go to the storage account.
  2. Open File Shares -> Users.
  3. Select your user’s folder.

Important notes:

  • It’s crucial to execute the cells sequentially instead of running all cells at once. Each code cell depends on the output of its previous cells.
  • Depending on your data volume, some cell execution may take a few minutes, so please be patient.
  • If you run into an issue, follow these preliminary steps:
    1. Sign off from AML workspace and sign in again.
    2. Restart the kernel.

Restart kernel

3. Rerun the notebook.

4. If that still doesn’t work, send me a direct message and make sure to tag me. Or you can always create a Support ticket and our team will assist you.

 

Check out the video below for a live demo!

 

 

Summary

I hope you find these notebooks useful. Give them a try and let us know what you think!

Got more scenarios where you would like to use a notebook for? We’d love to hear! You can reach us by sending me a direct message, or posting a comment below, or posting your feedback on Azure Sentinel feedback forums.

 

 

 

Updated Nov 03, 2021
Version 13.0
  • hugom7500's avatar
    hugom7500
    Copper Contributor

    Hello Ms.Nguyen,

    Thanks for your video and article on you use Azure Sentinel notebooks to detect credentials leaks. I follow your tuto but i didn't find ant credential leaks on my log analytics. So my question is : Do you know how to simulate a credential leaks to proof that the notebooks is efficiently ?

    Thanks in advance for your response.

    Best regards,

    Hugo M.

  • hugom7500 Somehow I missed your message. My apologies.

    To test notebook's efficiency, I would test it with some sample data that follows the credentials format I detailed in the blog (for example, Azure Storage Account Shared Access Signature). The sample data can be ingested into either a Log Analytics custom table, ADX cluster, or blob storage. Then run the notebook and you should expect to get the sample credentials in a csv file as the notebook output.