This installment is part of a broader series to keep you up to date with the latest features in Azure Sentinel. The installments will be bite-sized to enable you to easily digest the new content.
This blog is a collaboration between myself and my colleague, Sreedhar Ande.
What truly sets Azure Sentinel apart from other SIEM tools or other security solutions in the market is the extensive use of machine learning to fuel built-in analytics and custom machine learning models. These capabilities are the culmination of decades of research and experience protecting Microsoft services at massive scale by Microsoft security experts. As you might already be aware, Microsoft Ignite 2020 announcements highlighted some of the most recent innovations in this space.
We are delighted to announce that 32 new Fusion detections and Build Your Own Machine Learning framework are now available in public preview! Below has a recap of what these features are and how they work.
Fusion Detections
Using machine learning, Fusion detections combine low- and medium-severity alerts from Microsoft and 3rd-party security products into high-severity incidents. By design, these incidents are low-volume, high-fidelity, and high-severity. Here is an example of how a Fusion incident looks like in Azure Sentinel portal.
The main goals of Fusion detections can be summarized into two points.
For more details of how Fusion technology works behind the scene, please check out this excellent article by our colleague, Ram Shankar Siva Kumar.
Our Fusion team recently released 32 new Fusion detections in public preview, reaching a total of 90 Fusion incident types which are turned on by default in Azure Sentinel. These additional detections fall into eight scenario types.
MDATP + Palo Alto Network firewall:
AAD IP + MCAS:
Under Analytics blade in Azure Sentinel portal, in your Active Rules view, a built-in rule of Fusion rule type named “Advanced Multistage Attack Detection” is enabled by default for all Sentinel workspaces. You have the option to disable the rule any time. There is no extra cost to use this detection rule on top of the normal data ingestion and storage cost. All you need for the rule to work is to have your data connectors configured and data ingested correctly. To see what data connector sources are required for each Fusion incident type, please refer to the documentation.
To get step-by-step instructions about Fusion in Azure Sentinel, please refer to our documentation, which has been revamped with updated detection descriptions, now includes MITRE ATT&CK Tactics and Techniques, and is now organized by threat classifications for easier navigation.
Build Your Own Machine Learning (BYO-ML)
Many security organizations understand the value of machine learning for security, though not many of them have the luxury of professionals who have expertise in both security and ML. We designed the framework Build-Your-Own ML (BYO-ML) for security organizations and professionals to grow with us in their ML journey. Organizations new to ML, or without the necessary expertise, can get significant protection value out of Azure Sentinel's built-in ML capabilities.
ML detection models can adapt to individual environments and to changes in user behavior, to reduce false positives and identify threats that would not be found with a traditional approach. Azure Sentinel makes it easier for data scientists in these organizations to unlock these insights with a BYO-ML framework.
For organizations that have ML resources and would like to build customized ML models for their unique business needs, we offer the BYO-ML platform. The platform makes use of the Azure Databricks/Apache Spark environment and Jupyter Notebooks to produce the ML environment. It provides the following components:
Besides all this, you can bring your own ML models, and/or your own Spark environment, to integrate with Azure Sentinel.
For more details on BYO-ML platform, please check out this excellent blog by our colleague, Andi Comisioneru. For supported use cases, please refer to the documentation.
To build custom ML models on your data, you have two options.
Note: To use the Notebooks, you must first create an Azure Machine Learning (ML) workspace. For step-by-step instructions on how to create an Azure Machine Learning (ML) workspace, please refer to the documentation.
Apache Spark™ provides a unified environment for building big data pipelines. Azure Databricks builds on this environment, providing a zero-management cloud platform, holistically addressing the platform needed for data analysts to develop their custom ML based security analysis.
You can either bring your raw data directly to the Azure Databricks ML environment, via EventHub or Azure Blobs or you can use the capabilities provided with Azure Sentinel, to export the data from Azure Sentinel Log Analytics tables. Regardless of the export methods used for raw data, you can use the libraries provided by BYO-ML framework to import the scoring of the ML model back into Sentinel Log Analytics tables for further processing and creating incidents.
You can either set up a new Azure Databricks environment or use one already set up for other use. To set up a new Databricks environment, please refer to the quickstarts document (note that MMLSpark used by our algorithm requires Spark 2.4.5).
On Azure Sentinel roadmap, we plan to support Azure Synapse in addition to Azure Databricks as a BYO-ML development environment.
Get started today!
We encourage you to explore these machine learning innovations in the Azure Sentinel to detect and protect your organization from threats.
Try it out, and let us know what you think!
You can also contribute new Notebooks in Azure Sentinel. Get started now by joining the Azure Sentinel BYOML GitHub community.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.