Create, Edit, and Monitor Data Collection Rules with the Data Collection Rule Toolkit

*Thank you to Jing Nghik for assisting with the creation of this toolkit and to the Customer Connection Program for testing this solution.*

 

*This is going to be a long blog. I recommend reading it but alternatively there will be a video recording soon that will cover the workbook. TLDR: This workbook serves as a toolkit for data collection rules to make creating, editing, and monitoring DCRs in an environment easier. It is available today in the Workbooks Gallery within Microsoft Sentinel.*

 

It can be a little confusing when it comes to creating, monitoring, and modifying data collection rules from Azure Monitor. These components are split up between Azure Log Analytics, Azure Monitor, and Microsoft Sentinel. To address this, a new workbook has been developed in order to make interacting with data collection rules easier, cleaner, and more efficient. The workbook is broken up into 4 main tabs:

1. Identify Data Sources/Create New DCRs: This tab can be used to create new data collection rules. The experience is streamlined so users can click buttons and switches in order to configure what data will be ingested.
2. Monitor/Modify Existing DCR’s: This tab can be used to review all existing data collection rules for an environment. This allows users to see what is already configured, what data they are ingesting, and where that data is going. It will also highlight items such as if a data collection endpoint is being used in a DCR and if there is ingestion transformation applied. If needed, there is a section to modify the template of a selected rule.
3. Dataflow and Transformation: This tab can be used to break down a selected data collection rule in order to show the data source, transformation KQL if it is configured, and the destination of the data per stream.
4. Simple reporting: This tab will show a simple breakdown of the type of DCR, the events that are being brought in, and the amount of data that each item is contributing to in the workspace.
5. Useful Tools: This tab can be used to find useful workbooks and external tools that can assist with data collection rules, migration from MMA to AMA, and more.

 

Creating Data Collection Rules

 

Linux

 

The Linux options are Syslog and CEF. These buttons open the existing experience for making data collection rules through the wizard provided by Azure Monitor.

To create a new DCR:

1. Click on the Linux DCR button to expand the options.

1. Click on either the Syslog or the CEF button to open the creation wizard.
2. If using CEF, click on ‘create new collection rule’ option.

1. Fill out the key details for the DCR.
2. Assign resources that should be subscribed to the DCR.
3. Set the data that should be collected.
4. Click ‘review and create’.
5. When validation has passed, click create.

Windows

 

The Windows section is much deeper. The Windows section is broken up into categories that determine which event IDs will be collected. The categories are:

• NSA: Event IDs recommended by the NSA.
• MITRE: Event IDs that align with the MITRE tactics.
• Recommended: Recommended event IDs based on Microsoft documentation.
• File Path DCRs

 

Selecting a category will produce a preconfigured array of event IDs and options for modifying the array. These event IDs are being converted to xPath in the background via a KQL function. This xPath is used when deploying the DCR. The only category that is different will be the file path DCR as it leverages the existing UI for DCR's. 

If looking to manually add or exclude events, there is a section for manually adding/excluding events that will modify the xPath.

The number of distinct event IDs is shown with the array of the IDs. Currently, DCRs have a limit of 100 items within xPath. To assist with this, the tool detects when there are more than 100 events and will generate a second set of xPath and a second template to deploy. If more than 200, the same will be done with a third template.

 

The events and a description of what the events are can be found below this. This section allows users to see exactly what they will be ingesting when configuring the xPath.

 

If looking for additional logs to consider to ensure coverage, an active effort by the MSTIC team provides a list of events that are similar to events in security events. This data can be referenced to see where else this data can be ingested from.

 

Once the events have been picked, a workspace destination, data collection endpoint, and name is needed. Once this is all set, the template can be deployed.

 

To create a new Windows based DCR:

1. Click on the Windows DCR button.
2. Select a scenario to deploy.
3. Select an ingestion tier if desired.
4. Manually enter missing event IDs if needed.
5. Modify the settings for the scenario to change the event IDs.
6. Review the events that are going to be ingested.
7. Select a workspace in the ‘Deploy to Workspace’ drop down.
8. Select a data collection point in the ‘DCE’ drop down.
9. Enter a DCR name(s) for each DCR that may be needed.
10. Click on the deploy button.

 

Transformation

 

The ‘Table Transformation DCR’ button will expand an interface that assists with creating a rule. This interface lists:

• Workspace: Workspace that houses the table that should be modified.
• Available tables: All tables that are not populated by data via AMA.
• Schema: Available schema for the selected table.
• TransformQuery: KQL that will be used to transform the data as it is ingested.

The goal of this tab is to allow users to create table specific ingestion transformation rules without having to leave the workbook. To create a new transformation DCR:

1. Click on the Table Transformation DCR button.
2. Select the workspace that houses the table that should be modified.
3. Click on the table that should be modified.

4. Give the DCR a name.
5. Review the schema of the table.

6. Enter the transformation query in the ‘TransformQuery’ section.
   1. Optional: Use the workspace editor by clicking on the ‘Workspace Editor’ button to validate the KQL before pasting it into the TransformQuery section.
   2. Note: The table in the TransformQuery must always be ‘source’.

7. Once done, click on the ‘deploy’ button.

 

Custom Log

 

This button will just open the existing UI for creating a new custom table with a custom log DCR.

To create a new custom log:

1. Enter a table name.
2. Attach it to an existing DCR or create a new one.
3. Attach it to an existing data collection endpoint or create a new one.
4. Upload a sample of the log.
5. Use the editor in order to apply ingestion transformation for this custom log.
6. Once done, click review and create.

Essentials

 

The Essentials button provides options to deploy a DCR that contains the key event IDs for using UEBA, Windows based hunting queries, or Windows based analytic rules. The goal for this section is to provide a quick start up for the three core features of Microsoft Sentinel. This can be useful early in a deployment when the team is evaluating which other event IDs should be ingested. To create a new Essential DCR:

1. Click on the Essentials button to expand the options.
2. Select either UEBA, Analytic Rules, or Hunting.

3. Select an ingestion tier if desired.
4. Manually enter any event ID if needed.
5. Review the array of event IDs that will be deployed.

6. Select a workspace to attach the DCR to.
7. Select a DCE to attach the DCR to.
8. Provide a name for the DCR.
9. When ready, click the ‘deploy’ button.

 

 

Monitoring and Modifying DCRs

 

The third tab of the workbook allows for users to monitor and review existing DCRs in the environment. The goal is to centralize the DCRs and enable them to be modified without having to leave Microsoft Sentinel.

 

Monitoring

 

The workbook leverages the Azure Resource Graph to grab the existing DCRs and parse them out into a user friendly manner.

 

• Data collection rule: Link to the data collection rule. Clicking on the link will open the resource within Azure Monitor.
• Clipboard: Highlights the properties of the DCR, such as destination, configured sources, and transformation KQL.
• Rule type: Highlights the type of DCR (Windows, Linux, custom).
• Syslog/Windows/SecurityEvents: Highlights if the source is configured. The link listed will highlight the configured data collection for that source.
• Collection endpoint: Highlights if the DCR is attached to a DCE. The link listed will open the DCE within Azure Monitor.
• Ingestion Transform: Highlights if the DCR has ingestion time transformation confirmed.

If looking to make a copy, the ‘export template’ button will open the blade with the ARM template of the selected DCR. This template can be easily be redeployed as a new DCR or saved externally for future use.

 

Modifying

 

If looking to modify the existing DCR, the section includes a JSON editor for any selected DCR.

 

The editor will list the main body of the DCR that was clicked on in the first section. For changes:

Adding a Data Collection Endpoint:

If looking to point an existing DCR to a data collection endpoint, it can be manually entered. For this, it would appear as so:

{

“properties”: {

“immutableId”: “DCR IMMUTABLE ID”,

“dataCollectionEndpointId”: “AZURE RESOURCE ID PATH HERE”,

“dataSources”: {…..

 

Ingestion Time Transformation:

If looking to add ingestion time transformation, a DCE will need to be attached. If this is already done, the transformKql item will need to be entered. It will appear as so:

    "dataFlows": [

      {

        "streams": [

          "Microsoft-SecurityEvent"

        ],

        "destinations": [

          "WORKSPACE NAME"

        ],

        “transformKql”: “source | KQL QUERY BODY HERE”

      }

    ]….

If looking to verify the KQL, the workspace editor can be opened by clicking on the ‘write transformation KQL’ button. Once everything is ready to go, click on the ‘deploy update’ button.

 

Further Breakdown

 

The fourth tab allows for further review of a selected DCR. This tab dissects the selected DCR to highlight the source, streams, transformation KQL, and destination.

 

The goal of this tab is to help break down selected DCRs in order to better see the main components of it. This allows users to isolate a DCR of interest while being able to easily view the configuration.

 

DCR Reporting

 

The next tab is a simple reporting table that will highlight which events are being collected by DCRs and how much ingestion they are generating.  This report will cover Windows security events, Syslog, and custom logs.

 

The goal of the reporting is to highlight if there are more than one DCR's that are collecting an event and reporting it to the same workspace. Unfortunately, DCR templates do not track which machines are provisioned under them so it is not possible at this time to report which machines may be reporting the same event twice. The closest that can be done is via KQL.

 

Additional Tooling

The final tab of the tool lists potentially relevant and useful tools that exist today in relation to data collection. Tools like workbooks can be opened within the DCR Toolkit without having to leave. Additional tools such as a DCR library and MMA to AMA migration script can be used.

 

With that, the tool is covered. Begin utilizing this tool when looking to speed up data collection and DCR creation. For more information on everything covered in this toolkit, please refer to the public documents:

• Azure Monitor Agent overview - Azure Monitor | Microsoft Learn
• Collect events and performance counters from virtual machines with Azure Monitor Agent - Azure Monitor | Microsoft Learn
• Collect text logs with Azure Monitor Agent - Azure Monitor | Microsoft Learn
• Tutorial - Forward syslog data to Microsoft Sentinel and Azure Monitor by using the Azure Monitor agent | Microsoft Learn
• Data collection endpoints in Azure Monitor - Azure Monitor | Microsoft Learn
• Data collection transformations - Azure Monitor | Microsoft Learn
• Add or delete tables and columns in Azure Monitor Logs - Azure Monitor | Microsoft Learn