Microsoft Sentinel Notebooks Ninja Part 3: Overview of the Pre-built Notebooks - the Grand List

[Rod_Trent]

This installment is part of a broader learning series to help you become a Jupyter Notebook ninja in Microsoft Sentinel. The installments will be bite-sized to enable you to easily digest the new content. 

• Part 1: What are notebooks and when do you need them?
• Part 2: How to get started with notebooks and tour of the features
• Part 3: Overview of the pre-built notebooks and how to use them – this post
• Part 3.5: Using Code Snippets to build your own Sentinel Notebooks 
• Part 4: How to create your own notebooks from scratch and how to customize the existing ones
•  

KNOWLEDGE CHECK: And, once you've completed all of the parts of this series, you can take the Knowledge Check. If you score 80% or more in the Knowledge Check, you can expect your very own Notebooks Ninja participation certificate from us.

Through Part 1 and Part 2 of this Microsoft Sentinel Notebook Ninja series, we’ve discussed the concepts and activities to best become acclimated with Jupyter notebooks for Microsoft Sentinel. The next step in our process is understanding the value of having ready-made notebooks ready for use as part of the solution.

When a customer stands-up Microsoft Sentinel for the first time, there are a number of additional pieces of ready-to-use collateral that are provided out-of-the-box including Analytics Rules, Hunting queries, Connectors, Solutions, Workbooks – and – you guessed it – Notebooks.

The notebooks are mostly one of three types:

• Exploration notebooks. These are meant to be used as they are or with your own customizations to explore specific hunting and investigation scenarios. Examples of this type include the Entity explorer series. (“Entity” refers to items such as hosts, IP addresses, accounts, URLs, etc.)
• Simple How-To notebooks like the Get Started notebook.
• Sample notebooks. These are longer and are meant to be instructional examples following a real or simulated hunt or investigation.

The following Grand List represents the notebooks that are provided when setting up Microsoft Sentinel.

Microsoft Sentinel Notebooks – The Grand List

Notebook Name	Type	Description	Special Requirements
A Getting Started Guide For Azure Sentinel ML Notebooks	Getting Started	This notebook guides you through the basic steps of using notebooks for security analysis. It covers all the basic steps you need to understand to start using the notebooks provided with Azure Sentinel.	Data type(s): SigninLogs External service(s): VirusTotal, MaxMind, MSTICPy Package(s): MSTICPy
A Getting Started Guide For PowerShell AML Notebooks	Getting Started, PowerShell	This notebook takes you through the basics needed to get started with PowerShell notebooks that leverage Azure Sentinel data and APIs.	Data type(s): SecurityEvent
A Tour of Cybersec notebook features	Getting Started	This notebook takes you through some of the features of Azure Sentinel notebooks and MSTICPy. It's a good second notebook to explore after "A Getting Started Guid for Azure ML notebooks". It covers data queries, visualization, data analysis, enrichment with threat intelligence and pivot functions. It can be run against you Azure Sentinel workspace or standalone, with sample data.	Data type(s): SecurityEvent, SecurityAlert If no data types are available, sample data will be used. Package(s): MSTICPy
Configuring your Notebook Environment	Configuration	This notebook takes you through detailed setup of your settings for Azure Sentinel Notebooks and the MSTICPy library. It covers: Setting up your Python environment for notebooks (not required for AML notebooks), Creating and editing your msticpyconfig.yaml file, Understanding and managing you config.json file.	Package(s): MSTICPy
Credential Scan on Azure Blob Storage	Hunting	This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Blob Storage using Azure SDK for Python.	Package(s): MSTICPy
Credential Scan on Azure Data Explorer	Hunting	This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Data Explorer using Azure SDK for Python and KQL.	Package(s): MSTICPy
Credential Scan on Azure Log Analytics	Hunting	This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Log Analytics using Azure SDK for Python and KQL.	Data type(s): SecurityEvent Package(s): MSTICPy
Entity Explorer – Account	Hunting	Use queries and visualizations to help you assess the security state of an AAD/O365 account or an account on a local host. It includes examining Azure Active Directory (AAD) and Office365 activity for an account and identifying any related anomalous behavior. Allows you to correlate the IP addresses in related events with threat intelligence sources.​	Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, AAD, OfficeActivity Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2
Entity Explorer - Domain and URL	Hunting	Use queries and visualizations to help you assess the security state of a DNS host or domain name. Examine DNS requests to look for malicious DNS usage, such as DNS tunneling, as well as anomalous DNS name lookups. Query threat intelligence and domain ownership using WhoIS information for a specific DNS name, to get a better understanding of the domain name’s reputation.	Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, DNS, Syslog Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2
Entity Explorer - IP Address	Hunting	Brings together a series of queries and visualizations to help you assess the security state of an IP address. It works with both internal addresses and public addresses.	Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat External service(s): VirusTotal, Alienvault OTX, IBM Xforce
Entity Explorer - Linux Host	Hunting	This notebook brings together a series of tools and techniques to enable threat hunting within the context of a singular Linux host. The notebook utilizes a range of data sources to achieve this but in order to support the widest possible range of scenarios this Notebook prioritizes using common Syslog data.	Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat Package(s):  kqlmagic, msticpy, pandas, pandas_bokeh, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2
Entity Explorer - Windows Host	Hunting	Brings together a series of queries and visualizations to help you determine the security state of the Windows host or virtual machine that you are investigating. It looks for related alerts, allows you to examine logon sessions and processes, check process command lines for IoCs and explores network traffic between the host and external endpoints.	Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat Package(s): kqlmagic, msticpy, pandas, numpy, matplotlib, bokeh, networkx, ipywidgets, ipython, scikit_learn, dnspython, ipwhois, folium, maxminddb_geolite2
Guided Hunting - Anomalous Office365 Exchange Sessions	Hunting	Brings together a series of data science techniques to help you hunt for anomalous sessions in your Office Exchange logs. It queries the OfficeActivity table, creates sessions from the PowerShell cmdlets (e.g. Set-Mailbox), trains a model and then visualizes the sessions.	Data type(s): OfficeActivity Package(s):  msticpy, pandas, kqlmagic
Guided Hunting - Base64-Encoded Linux Commands	Hunting	This notebook is a collection of tools for detecting malicious behavior when commands are Base64-encoded. It allows you to specify a workspace and time frame and will score and rank Base64 commands within those bounds. It utilizes multiple data sources, primarily focusing on Azure Sentinel Syslog data augmented by telemetry from the MSTIC research branch of the AUOMS audit collection tool.	Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, folium, maxminddb_geolite2, BeautifulSoup
Guided Hunting – Covid-19 Themed Threats	Hunting	Brings together a number of techniques and approaches to allow an analyst to hunt for indicators of Covid-19 themed threats within an organization's data. This includes hunting templates for network, cloud and host logs in order to find potentially malicious Covid-19 themed activity.	Data type(s): SecurityEvent, OfficeActivity, CommonSecurityLog Package(s):  MSTICPy
Guided Investigation - Anomaly Lookup	Investigation	Gain insights into the possible root cause of an alert by searching for related anomalies on the corresponding entities around the alert’s time. This notebook will provide valuable leads for an alert’s investigation, listing all suspicious increase in event counts or their properties around the time of the alert, and linking to the corresponding raw records in Log Analytics for the investigator to focus on and interpret.	Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Auditd_CL, OfficeActivity, CommonSecurityLog Package(s):  MSTICPy
Guided Investigation - Process Alerts	Investigation	This notebook is intended for triage and investigation of security alerts. It is specifically targeted at alerts triggered by suspicious process activity on Windows hosts. Some of the sections will work on other types of alerts but this is not guaranteed.	Data type(s): SecurityEvent External service(s): OTX, VirusTotal, XForce
Guided Investigation - SolarWinds Post	Investigation	This notebook assists defenders in hunting for SolarWinds post compromise Tactics , Tools and Procedures (TTPs) across different environments both on-premises and cloud data sources.	Data type(s): SecurityEvent, AuditLogs, SigninLogs, OfficeActivity External service(s): OTX, VirusTotal, XForce
Guided Investigation – Alert Triage	Investigation	Rapidly triage alerts raised by a range of sources (Azure Sentinel, MDE, MCAS, ASC, etc.) by enriching alert data with Threat Intelligence and OSINT. This notebook is intended to be used by analysts as part of a standard operating procedure. The notebook uses UI widgets and simple workflow to ensure an easy introduction to notebooks for all levels of analyst levels. This Notebook uses MSTICpy to connect to Threat Intelligence sources	Data type(s): SecurityAlert Package(s):  MSTICPy
Azure Synapse - Configure Azure ML and Azure Synapse Analytics	Getting Started, Configuration, Synapse	This notebook provides step-by-step instructions to set up Azure ML and Azure Synapse Analytics environment for your big data analytics scenarios that leverage Azure Synapse Spark engine. It covers: Configuring your Azure Synapse workspace, creating a new Azure Synapse Spark pool, configuring your Azure Machine Learning workspace, and creating a new link service to link Azure Synapse with Azure Machine Learning workspace. Additionally, the notebook provides the steps to export your data from a Log Analytics workspace to an Azure Data Lake Storage Gen2 that you can use for big data analytics.
Azure Synapse - Detect potential network beaconing using Apache Spark	Hunting, Synapse	In this sample guided scenario notebook, we will demonstrate how to set up continuous data pipeline to store data into azure data lake storage (ADLS) and then hunt on that data at scale using distributed processing via Azure Synapse workspace connected to serverless Spark pool. Once historical dataset is available in ADLS, we can start performing common hunt operations, create a baseline of normal behavior using PySpark API, and also apply data transformations to find anomalous behaviors such as periodic network beaconing as explained in the blog - Detect Network beaconing via Intra-Request time delta patterns in Microsoft Sentinel - Microsoft Tec....	Data Type(s):   CommonSecurityLogs Prerequisite -  Please run configuration notebook: Azure Synapse - Configure Azure ML and Azure Synapse Analytics first.
Guided Web Shell Investigation - MDE Sentinel Enrichments	Investigation	This notebook investigates Microsoft Defender for Endpoint (MDE) web shell alerts. The notebook will guide you through steps to collect MDE alerts for web shell activity and link them to server access logs to identify potential attackers.	Data type(s): SecurityAlert, W3CIISLog Package(s):  MSTICPy
Hands on - Data Discovery using Azure REST API	Getting Started	This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel data discovery by using Azure REST API.	Data type(s): SecurityEvent
Hands on - Surfing Your Data using Azure SDK for Python	Getting Started	This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel log data discovery by using Azure SDK for Python and Kusto Query Language (KQL).	Data type(s): SecurityEvent
Machine Learning in Notebooks Examples	Getting Started	This notebook template guides you through using time series analysis to detect anomalous network activity, clustering to highlight unusual logon sessions, and using Markov Chain to identify anomalous sequences in events.	Data type(s): SecurityEvent Package(s):  MSTICPy

These notebooks and more also exist in the official Microsoft Sentinel GitHub repository.  This repository contains notebooks contributed by Microsoft and the community to assist hunting and investigation tasks in Microsoft Sentinel. There are a number of notebooks in the GitHub repository that are valuable additions, but are not supplied automatically when you stand-up Microsoft Sentinel. You should regularly review this repository for more, or use the RSS feed to be notified of additions and changes.

Notebooks, like all other components and features for Microsoft Sentinel, are under constant review and undergoing constant improvement. Improvements and changes come from feedback and suggestions from our customers. You can use the DL (asinotebooks@service.microsoft.com) to send your questions, issues, and feedback and our various product teams will monitor and respond.

As these updates are made available, we’ll update the Grand List.

We are super-excited to be bringing this blog series and the training to you! To register, visit https://aka.ms/SecurityWebinars, look for Microsoft Sentinel | Become a Notebooks ninja webinar and fill out the registration form.

Look for more great knowledge on Microsoft Sentinel Notebooks as we prepare to deliver Part 4 of this series: How to create your own notebooks from scratch and how to customize the existing ones.

More reading/tutorial resources:

• Notebooks at the Azure Sentinel GitHub Repository
• Use Jupyter Notebook to hunt for security threats
• Detect Credential Leaks using Azure Sentinel Notebooks! (video)
• Azure Sentinel: Notebooks - Getting Started (video)
• Detect Malicious Base64-Encoded Commands on Linux Hosts (video)
• Microsoft Sentinel Weekly Newsletter