Co-author - Jeremy Tan
In today's rapidly evolving cybersecurity landscape, staying ahead of threats is crucial. Microsoft Defender's custom detection rules offer a powerful way to proactively monitor and respond to security threats. These user-defined rules can be configured to run at regular intervals to detect security threats—generating alerts and triggering response actions when threats are detected.
If you are a Microsoft Sentinel user and have connected your Sentinel workspace to Microsoft Defender, you are probably more familiar with analytics rules in Microsoft Sentinel and are looking to explore the capabilities and benefits of custom detections. Understanding and leveraging custom detection rules can significantly enhance your organization's security posture. This blog will delve into the benefits of custom detections and showcase scenarios that highlight their capabilities, helping you make the most of this robust feature.
We are excited to release these brand-new enhancements that are now available in public preview.
What’s new in custom detections?
The improvements in custom detections aim to enhance their functionality and usability, making it easier to manage and respond to security threats effectively.
- Unified user defined detection list:
- Manage all your user-defined detections from Microsoft Defender XDR and Microsoft Sentinel in one place.
-
- Filtering capabilities for every column.
-
- Search freely using rule title or rule ID.
-
- View the new workspace ID column (filterable) for multi-workspace organizations that onboarded multiple workspaces to the unified SOC platform.
-
- Manage all detections from MTO portal across all your tenants.
-
- Show details pane for every rule (whether custom detection or analytics rule).
-
- Perform the following actions on rules:
-
-
- Turn on/off
- Delete
- Edit
- Run (only for custom detections)
- Open rule’s page (only for custom detections)
-
-
- Migrate eligible scheduled custom detections to near real-time custom detections with one click using the new migration tool.
- Dynamic alert titles and descriptions: Dynamically craft your alert’s title and description using the results of your query to make them accurate and indicative.
- Advanced entity mapping: Link a wide range of entity types to your alerts.
- Enrich alerts with custom details: Surface details to display in the alert side panel.
- Support Sentinel-only data: Custom detections support Microsoft Sentinel data only without dependency on Microsoft Defender XDR data.
- Flexible and high frequency support for Sentinel data: Custom detections support high and flexible frequency for Microsoft Sentinel data.
The benefits of custom detections
Let’s examine some of the key benefits of custom detections:
- Query data from Defender XDR and Sentinel seamlessly: You can create custom detection rules that query data from both Microsoft Sentinel and Defender XDR tables seamlessly, without the need of sending Defender XDR data to Sentinel.
- Cost efficiency: Save on ingestion costs if you don’t need to retain Microsoft Defender XDR data in analytics tier for more than 30 days but have detection use cases involving both Defender XDR and Sentinel data.
- Detect threats immediately and remove dependency on quick ingestion: near real time (NRT) custom detections monitor events as they stream, while standard custom detections evaluate both the event ingestion time and the time the event was generated.
- Unlimited NRT detections: NRT custom detections are unlimited, you can create as many as you need. Since they are based on a streaming technology, they are not generating any load on the system.
- Native remediation actions: You can configure custom detection rule to automatically take actions on devices, files, users, or emails that are returned by the query when your detection query is correlating Defender XDR and Microsoft Sentinel data, or Defender XDR data only.
- Entity mapping: Entities are automatically mapped to the alert for all XDR tables.
- Out of the box alert de-duplication: To reduce alert fatigue when alert generated with the same impacted entities, custom details, title and description - they will merge to the same alert (keeping all raw events linked to the single alert). With this capability you don’t need to worry about duplicated alerts – we take care of it for you.
- Built-in functions: You can leverage built-in enrichment functions when you build your custom detection queries, such as FileProfile(), SeenBy(), DeviceFromIP() and AssignedIPAddresses().
- Extended lookback period: Custom detections have a long lookback period of up to 30 days for rules that run once a day, ideal for historical trending detections.
Common scenarios
To truly understand the power and versatility of custom detection rules in Microsoft Defender, it's essential to see them in action. In this section, we'll explore several common use cases that demonstrate how these new capabilities can be leveraged to enhance your organization's security posture. These scenarios highlight the benefits of the capabilities, providing you with actionable insights to implement in your own environment.
Use Case – detecting potential malicious activity
In this use case, we aim to detect potential malicious activity by monitoring logon attempts from different IP addresses. We will implement a custom detection rule that:
- Monitors successful logon by a user from one IP address and a failed logon attempt from a different IP address (may indicate a malicious attempt at password guessing with a known account).
- Enriches alerts with user's information from Microsoft Defender for Identity’s IdentityInfo table, including Job title, Department, Manager’s name, and assigned roles.
- If the user has been found in the 'Terminated Employees’ watchlist, indicating that the user has been notified for termination or marked as terminated, reflect this in the alert name and description.
- Runs once a day with a lookback period of 30 days, avoiding duplicate alerts on subsequent intervals.
Let’s walk through the creation of the custom detection rule and examine the outcome.
1. Here is the sample KQL query we will run in advanced hunting page to create the custom detection.
let logonDiff = 10m;
let Terminated_Watchlist =
_GetWatchlist("TerminatedEmployees")
| project tolower(SearchKey);// Get the TerminiatedEmploees Watchlist
let aadFunc = (tableName:string)
{
table(tableName)
| where ResultType == "0"
| where AppDisplayName !in ("Office 365 Exchange Online", "Skype for Business Online") // To remove false-positives, add more Apps to this array
| extend SuccessIPv6Block = strcat(split(IPAddress, ":")[0], ":", split(IPAddress, ":")[1], ":", split(IPAddress, ":")[2], ":", split(IPAddress, ":")[3])
| extend SuccessIPv4Block = strcat(split(IPAddress, ".")[0], ".", split(IPAddress, ".")[1])
| project SuccessLogonTime = TimeGenerated, UserPrincipalName, SuccessIPAddress = IPAddress, SuccessLocation = Location, AppDisplayName, SuccessIPBlock = iff(IPAddress contains ":", strcat(split(IPAddress, ":")[0], ":", split(IPAddress, ":")[1]), strcat(split(IPAddress, ".")[0], ".", split(IPAddress, ".")[1])), Type
| join kind= inner (
table(tableName)
| where ResultType !in ("0", "50140")
| where ResultDescription !~ "Other"
| where AppDisplayName !in ("Office 365 Exchange Online", "Skype for Business Online")
| project FailedLogonTime = TimeGenerated, UserPrincipalName, FailedIPAddress = IPAddress, FailedLocation = Location, AppDisplayName, ResultType, ResultDescription, Type
) on UserPrincipalName, AppDisplayName
| where SuccessLogonTime < FailedLogonTime and FailedLogonTime - SuccessLogonTime <= logonDiff and FailedIPAddress !startswith SuccessIPBlock // Compare the success and failed logon time
| summarize FailedLogonTime = max(FailedLogonTime), SuccessLogonTime = max(SuccessLogonTime) by UserPrincipalName, SuccessIPAddress, SuccessLocation, AppDisplayName, FailedIPAddress, FailedLocation, ResultType, ResultDescription, Type
| extend Timestamp = SuccessLogonTime
| extend UserInTerminatedWatchlist = iif(UserPrincipalName in (Terminated_Watchlist), 'True', 'False') // Check if the impacted user is found in the Watchlist
| extend AlertName = iif(UserInTerminatedWatchlist == 'True', "Successful logon by a 'Terminated Employees Watchlist' user from one IP and a failed logon attempt from a different IP","Successful logon from IP and failure from a different IP") // This is the define the dynamic alert value
| extend AlertDescription = iif(UserInTerminatedWatchlist == 'True', "A Successful logon by a 'Terminated Employees Watchlist' user onto an Azure App from one IP and within 10 mins failed to logon to the same App via a different IP (may indicate a malicious attempt at password guessing with known account). ","A user account successfully logs onto an Azure App from one IP and within 10 mins failed to logon to the same App via a different IP (may indicate a malicious attempt at password guessing with known account).") // This is to define the dynamic alert description
| extend UserPrincipalName = tolower(UserPrincipalName)};
let aadSignin = aadFunc("SigninLogs");
let aadNonInt = aadFunc("AADNonInteractiveUserSignInLogs");
union isfuzzy=true aadSignin, aadNonInt
| extend Name = tostring(split(UserPrincipalName,'@',0)[0]), UPNSuffix = tostring(split(UserPrincipalName,'@',1)[0])
| join kind=leftouter (
IdentityInfo // Correlate with IdentityInfo table
| summarize arg_max (TimeGenerated,AccountObjectId, Department, JobTitle, Manager, AssignedRoles, ReportId, IsAccountEnabled) by AccountUpn
| extend UserPrincipalName=tolower(AccountUpn)
) on UserPrincipalName
2. On the top right corner of the advance hunting page, select ‘create custom detection’ under Manage rules.
3. Populate the relevant rule’s information.
4. Specify alert title and description by referencing the AlertName and AlertDescription fields defined in the query, as we will dynamically craft the alert title and description, depending on whether the impacted user is found in the 'Terminated Employees’ watchlist.
5. In the entity mapping section, you will find some entity mappings that we have pre-populated for you, which would save you some time and effort. You can update or add the mappings as you wish.
6. Let’s add some additional mappings. In this example, I will add IP entities under Related Evidence.
7. In the Custom details section, I will add the following key-value pairs to surface additional information of the impact user in the alert.
8. On the Automated actions page, because we are correlating Sentinel data with Defender XDR table (IdentityInfo), you have the option to select first-party remediation actions, which is ‘Mark user as compromised’ in our case.
9. Review the configuration of the rule and click Submit.
10. Now, let’s examine how the incident/alert would look. Below is a sample incident triggered.
11. Select the alert and you will find the custom details on the right pane, surfacing additional information such as Job title, Department, Manager’s name and Assigned roles that we configured.
12. The impacted user from the above incident was not found in the 'Terminated Employees’ watchlist. Now, let’s examine how the incident/alert would look when the impacted user is found in the watchlist.
13. In my environment, I have configured the watchlist and will be using ‘MeganB’ for simulation.
14. Notice how the alert title and description is different from the one generated earlier, to reflect user found in the watchlist.
15. The rule will run once a day with a look back period of 30 days. However, custom detection will not create duplicate alerts if the same impacted entities are found in the subsequent runs. Instead, you will find the Last activity time being updated and more events showing up in the result table of the alert page.
Conclusion
Custom detection rules in Microsoft Defender offer a powerful and flexible way to enhance your organization's security posture. By leveraging these user-defined rules, you can proactively monitor and respond to security threats, generating detailed and actionable alerts. The recent enhancements—such as unified detection lists, dynamic alert titles, and advanced entity mapping—further improve the functionality and usability of custom detections.
Ready to enhance your threat detection capabilities? Start exploring and implementing custom detection rules in Microsoft Defender today to safeguard your digital assets and maintain a strong security posture.
Useful links
Microsoft