- Home
- :
- Microsoft Security and Compliance
- :
- Azure Sentinel
- :
- The FAQ companion to the Azure Sentinel Ninja training
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Home
- :
- Microsoft Security and Compliance
- :
- Azure Sentinel
- :
- The FAQ companion to the Azure Sentinel Ninja training
01-03-2021
12:03 AM
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Email to a Friend
- Printer Friendly Page
- Report Inappropriate Content
01-03-2021
12:03 AM
(Updated Jan 26th 2021)
While extensive, the Ninja training has to follow a script and cannot expand on every topic. Like any training, you may have questions after the session. This live blog post tries to address that by providing answers to common questions ordered by the Ninja training modules.
Let go!
Module 1: Get started with Azure Sentinel
Q: How do I do a free of charge trial for Azure Sentinel?
There is no straight forward free trial for Sentinel:
- Every new workspace is not billed for *Azure Sentinel* for a month.
- However, the Azure Sentinel cost is made of the Azure Sentinel cost and the Log Analytics cost, and there is *no free trial for Log Analytics*.
There is, however, some usage that is always free, and you try to limit yourself to those to have a free POC:
- Log Analytics is free for the first 5GB for each month, across an *account*
- Both Log Analytics and Sentinel are free when Sentinel is deployed for selected sources such as Office 365.
So, how do I run a free PoC? Either of those:
- Using free sources only.
- On top of an existing, already paid for Log Analytics data. Giving 30 days of free Sentinel ingestion.
- A dedicated Azure tenant unrelated to the EA gives 30 days of free Sentinel ingestion and 5GB/m free Log Analytics ingestion. The 30 days can be restarted by creating a new workspace.
Q. How can I send sample data?
For CEF (CommonEventLog) events stored in a file, you can use Logstash to read data from your CEF sample log file and send it directly into the Log Forwarder.
This is the Logstash sample config file:
{
input {
file {
path => "/home/stefan/samplelogs/cef.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
output {
# change to your log forwarder host and port
tcp {
host => "127.0.0.1"
port => 514
}
}
Q: How can I have a direct link to the Azure Sentinel overview page? Any other page?
You don't need to get to Azure Sentinel through the Azure Portal every time. Just bookmark any page (or copy the URL) and use it to access your favorite starting point. The URL will have the following format, with the blade number changing based on the specific page you wanted to start with:
Pricing and billing
Q: How do I know how much I am charged?
Use the Azure portal cost management screen. Filter by the scope relevant to you (the workspace or resource group, for example).
Q: How do I know which sources contribute to my bill?
The usage information is available in the workspace, and you can use these queries to report or as a starting point for your reporting. The usage reporting workbooks for Azure Sentinel uses this information to provide a comprehensive view of usage.
Q: If a workspace ingests data and has Azure Sentinel is enabled, is all data consumed by log analytics into that workspace counted as being ingested by Sentinel, or is it only data that Sentinel connectors are enabled on?
Azure Sentinel bills for all data ingested to the workspace apart from free sources. The Sentinel connectors gallery is only one way to connect data to Azure Sentinel. If you connected using other means such as Azure diagnostics logs, custom sources, or additional agent streams, it is still Sentinel data.
While the built-in analytics may cover just some sources (and covers sources not explicitly listed in the connector gallery), users can create custom analytics on any data in the workspace.
Q: If I retain logs for longer than the included 90 days, do they pay for retention for sources that are free to ingest, such as Office Activity?
Our official pricing is to charge for retention beyond 90 days for sources ingested for free. However, you may find that in some cases, we do not actually charge. While we may start charging for such retention in the future, we will not charge for past charges not collected.
Q: Does collecting logs across regions incur networking charges?
Network communication between regions in Azure costs money, and the question is, how does this relate to Azure Sentinel?
Telemetry collected using an agent, the Log forwarder, or custom connectors using the ingest API, if the relevant source is not in the workspace region, would incur inter-region bandwidth costs.
However, service to service connectors, including Azure diagnostics sources, Office 365 and Microsoft 365 sources will not incur such costs even if the telemetry source is in a different region than the workspace. For example, if you collect telemetry from an Azure Firewall, there is no bandwidth charge regardless of the firewall region.
Please note that the Azure Sentinel documentation is incorrect and identity several agent-based sources such as DNS and Windows Firewall as service to service connectors.
Q: When I enable Azure Sentinel on an existing Log Analytics workspace, how does pricing change?
If you enable Azure Sentinel in an existing Log Analytics workspace:
- There will be an additional cost for Sentinel applied to all data in the workspace.
- All data will be retained for 90 days with no additional charge. Additional retention remains at the current Log Analytics rate.
- The sources free for Sentinel ingestion will not be charged for ingestion (Log Analytics or Sentinel tier). Retaining this data beyond 90 days costs at the Log Analytics retention price.
Q: Can Azure Sentinel capacity reservations be reserved for 1 year, 3 years?
No. Azure Sentinel capacity reservations are different from Azure reserved instances and behave like standard Azure meters, billed daily. They differ from pay-as-you-go pricing as they offer a lower per-unit price for reserving a larger amount of units.
Q: Why is the pricing calculator using different capacity reservations for Log Analytics and Azure Sentinel?
Since capacity reservation is at 100GB/d increment, at one point between 0 and 100, it makes sense to commit to the higher capacity. So if you only need 10GB/d, you will use pay-as-you-go pricing, while if you need 90 GB/d, you will commit to 100 GB/d. Since the discount level is different for Log Analytics (up to 25%) and Azure Sentinel (up to 60%), the cutoff value you would like to commit to the full additional 100GB/d is different.
Q: How much is the data compressed when stored?
The internal implementation is not relevant. Billing is based on the ingest, uncompressed volume.
Module 2: How is Azure Sentinel used?
Azure Sentinel as part of the Microsoft Security stack
Q: On a Windows system with Defender for endpoints already installed, would you install the Log Analytics agent to report Security Events to Azure Sentinel as well?
In general, the answer is yes, but it would depend on the use cases. Windows events are wide in scope but broadly fall into two groups:
- Activity (such as process, file, and network activity) that overlap with MDATP.
- Management audit (for example, user management) is not in the MDATP domain.
Q: How does Azure Sentinel compare with the Graph Security API?
The Azure Sentinel and Graph Security API teams work very closely. Sentinel utilizes the Graph Security API when applicable, for example, to get threat intelligence or integrating with SIEM and ticketing systems.
The main difference is that the Graph Security API does not support raw telemetry, which is the bread and butter of Azure Sentinel. The Sentinel connectors focus on getting raw telemetry. There are exceptions in areas we need to improve the cross-utilization, and we are working on that.
Side by side with your existing SIEM
Q: How do I forward alerts from Azure Sentinel to another system?
See the Ninja training side-by-side section.
Q: How do I forward data, alerts, or events from my current SIEM to Azure Sentinel?
The most common way would be to use Syslog or CEF, which most SIEM products support. Note that you would like to forward from the 3rd party SIEM collector layer in many cases, which is more efficient than overloading the 3rd party SIEM processing layer.
The following links can get you started:
- Splunk:
- ArcSight (CEF),
- Kiwi Syslog Server (Syslog),
- SolarWinds (Syslog)
Q: Ticket System Integration? Is it ServiceNow only?
While ServiceNow is the most popular ticketing system and many of our examples are focused on it, Logic Apps, on which the integration is based, has connectors with other ticketing systems:
If not available, you can still connect to your ticketing systems using a custom Logic App connector, the HTTP connector that supports most APIs, or an Azure function from Logic Apps.
Q: How do I forward events from Azure Sentinel to another SIEM?
We do not recommend forwarding all events from Azure Sentinel to your on-prem SIEM. It may imply you are not getting enough value from Azure Sentinel and worth looking into.
In case you want to forward events (all of some), export from Azure Sentinel / Log Analytics to Azure Storage and Event Hub or move Logs to Long-Term Storage using Logic Apps.
Module 3: Workspace and tenant architecture
Q: Best practice is to minimize the number of workspaces, but I want to split the bill. How do I do that?
Read how to report on the ingestion volume per computer, resource, resource group or subscription.
Q: Are the best practices for Log Analytics and Azure Sentinel concerning workspace architecture the same?
Not always. Log Analytics and Azure Sentinel have different use cases and users, which sometimes require a different approach. If Azure Sentinel uses a workspace, use the Azure Sentinel best practices. Also, try to minimize the amount of data not relevant to Azure Sentinel in the workspace to avoid unnecessary costs.
As a reference, you can find the Log Analytics multi-workspace best practices here:
Q: Should I use a workspace in a region geographically close?
Apart from regulatory requirements, the geographical location of workspaces does not make a difference. Specifically, the latency between regions does not influence Azure Sentinel services in a meaningful way. This may imply that you should pick your region based on price if there are no other requirements.
Q: Can I move the Azure Sentinel workspace to a different Resource Group of subscription?
While the feature is available for a Log Analytics workspace, we have not comprehensively tested moving an Azure Enabled workspace to a new subscription. Customers have done it before, and the one issue we encountered was that analytics rules do not work anymore, disabling and enabling the rules help. That said, there might be other issues, so the prudent solution would be to start over.
Module 4: Collecting events
General
Q: What is the collection latency for events collected by Azure Sentinel
The latency is different for different sources and mostly stems from the source behavior, with Azure Sentinel (and Log Analytics) adding very little. Azure AD and Office 365 do not provide real-time events and have a typical latency of 30 minutes with longer delays at times. This delay would be experienced in Azure Sentinel or any other SIEM collecting events from those sources.
You can read more on the topic, including how to measure the delay, here.
Log Forwarder
Note that the Log Forwarder is based on the Linux based Log Analytics Agent (MMA), so the questions in the next section, as far as they pertain to the Linux MMA, are relevant for the Log Forwarder as well.
Q: How do I set the Log Forwarder to listen to encrypted Syslog
Configure the Syslog server part of the Log Forwarder (rsyslog or Syslog-NG) to listen to TLS based Syslog:
Q: Can I filter Syslog of CEF events?
Yes, See the Log Forwarder webinar: YouTube, MP4, Deck.
Q: Should I filter firewall events?
Unlike windows events, Firewall events are simple and of only a handful of types. The most common event types (using Palo Alto's terminology) are:
- Traffic events - any connection through the Firewall.
- Threat events - any URL accessed through the Firewall (the name is misleading here)
Both have significant value for your security but have a large volume and therefore cost. Preferably, all should be collected. Inbound failures are candidates for filtering out, as they include a huge volume of low quality attack attempts.
Q: What size VM should I use for the Log Forwarder?
The Log Forwarder does little itself as parsing is done in the cloud. Therefore, comparatively, smaller and cheaper systems can be used.
You can find official sizing information in the documentation.
In addition, recent reports from customers have suggested:
- 500 GB/d of CEF data using a three VM scale set of Standard_D4s_v3 (4 CPU, 16GB) VMs.
- 6000 EPS of CEF data using a single physical VM: 8 vCPUs, 16 GB memory, Intel Xeon Platinum 8171M CPU @ 2.60GHz.
Use a VM scale set with an Azure load balancer or an on-prem load balanced to go beyond.
Log Analytics Agent and Azure Monitor Agent
Q: Is the workspace key stored on the agent machine?
We don't store the workspace key. It's only used during onboarding to generate the certs used for on-going communications by the Agent. The Workspace ID is stored in a config file per workspace here: /etc/opt/microsoft/omsanget/ws-id.
Q: Can Azure Sentinel filter Windows Events?
The Log Analytics agent (MMA) offers limited control over the Windows events forwarded. You can set a collection tier for all agents. However, the common tier is often not enough for Azure Sentinel customers, especially as it has to be set for all agents.
The new Azure Monitoring Agent (AMA) can granularly filter Windows events using WEF like XPath expressions.
Q: Does the Agent compress data from on-prem to the cloud?
Yes, the Log Analytics agent (MMA) compresses data when sending it to the cloud. This is used for Syslog, CEF, and local Windows or Linux telemetry. For Linux, the agent uses Zlib compression. The lib compression ratio is typically between 2:1 to 5:1 and maxes out theoretically at 1032:1
Q: Are there limits to how much custom logs (i.e. files) the Log Analytics agent can collect
The Log Analytics agent can collect files located on the machine it is installed on. This feature is intended for collecting local files and not as a means for aggregated collection, for example replacing Syslog. It is therefore limited to 500 EPS (Events, or log lines, per second) and exhibits issues if attempting to collect and forwards higher rates. A common issue that happens at higher rates is event duplication. If you need to collect files at a high volume into Azure Sentinel, consider using Logstash as described here.
Specific connectors
Q: Which API does Azure Sentinel to collect CloudTrail events
The AWS CLoudTrail API LookupEvents end point.
Q: Missing DNS Lookups Data
First, try resetting the config or just loading the configuration page once in the portal. For resetting, just change a setting to another value, then change it back to the original value, and save the config (Source).
If this does not work, there might be a caching problem in the agent, requiring deleting the Health Service State folder. Use these steps:
- Start an Administrative Command Prompt and run 'Net Stop HealthService'
- Start File Explorer and navigate to C:\Program Files or C:\Program Files(x86)
- Go to this location: Microsoft Monitoring Agent\Agent
- Rename the folder Health Service State to Old Health Service State
- In the Administrative Command Prompt, run Net Start Health Service
Module 5: Log Management
Q: The log search is limited to 10K results; what can I do?
Indeed, there is a 10K cap on the result set size in the UI. There is usually not meaningful need to review so many results in the UI. The API, and hence PowerShell, can return up to 500,000 results. Use the PowerShell script to run a query and get the results in a CSV file.
If you still need more than 10K results in the portal:
- You can transform your results into an array, which can hold much more than 10K values.
- Reduce the size of your results - you can use "distinct Computer," "summarize by Computer," or "summarize make_set" to remove duplicate values from your results (Also, if all you need is that computer's name, "project" only that column)
Q: Which columns are displayed in a search result if not specifically projected?
Multiple heuristics determine which fields to display. Some common ones are:
- Hiding system columns that typically pollute the visual space and are not commonly used (_ResourceId, for example)
- Hiding any columns that do not contain any data for the entire result set
-
Hiding by default predefined columns for specific tables.
Q: Can I delete unused custom log tables from a workspace?
The tables will disappear once empty. Use the purge API or wait for the retention period ends.
Q: How much is the data compressed when stored?
The internal implementation is not relevant. Billing is based on the ingest, uncompressed volume.
Q: Are there any standard fields available for each record?
Standard fields include event time fields, record type, and billing information fields. See Standard properties in Azure Monitor Logs for more details.
Module 6: Enrichment: TI, Watchlists, and more
Q: How often does Azure Sentinel Poll TAXII for new IOCs, and can this be configured?
This depends on the TAXII server. Generally speaking, if a well-formed TAXII server adheres to the standards, the TAXII data connector will pull the entire collection on the first connection and then pull only incremental changes every minute.
Q: What information from the TAXII server does Azure Sentinel pull
Currently, Azure Sentinel requests from the TAXII server and ingests only indicator STIX objects. We are planning the support of other STIX Domain Objects in the future. We perform a mapping from STIX to the ThreatIntelligenceIndicator table schema when we import the data.
Q: Is pagination supported in TAXII?
Yes, we support pagination. The TAXII server determines the size of the page. The TAXII server that you are connected to decides the number of IOC's to be returned in a request.
Q: Do we have specific IP addresses that we would use to pull this data into Sentinel?
While there are no specific IP addresses, they will be Azure IP addresses within the relevant workspace region.
Q: Since the Graph Security API is a tenant level, can one control what threat indicators each workspace receives?
The Graph Security API operates on a Tenant level. So operations performed against the Graph are based on your AAD tenant. What this means for the TI APIs is when you send threat indicators to Graph API with a target product of Azure Sentinel (or Defender for endpoints), you are supplying those threat indicators to your tenant, your entire organization.
Any Azure Sentinel workspace that connects the Threat Intelligence – Platforms data connector will tap into this tenant-level repository of threat indicators.
To send threat indicators to Graph API, the sending application (the app supplying the threat indicators) must be granted the proper permissions to write indicators to the Graph API on behalf of the tenant. This is a highly privileged operation that requires a Global Admin level user to consent on behalf of the application. Organizations generally restrict this ability and do not grant such permissions to applications for testing. For example, at Microsoft, I (Jason) cannot configure any application to send threat indicators to the Graph API on behalf of the Microsoft tenant.
If an organization is experiencing a problem, it means that a Global Admin has authorized the application providing incorrect TI or test data to push threat indicators into Graph on behalf of their tenant. It might be needed to revisit this decision.
Q: How do I use the confidence score associated with threat intelligence IoCs?
The confidence score is meant to convey the level of certainty the provider of the threat indicator feels the observations of the pattern in the indicator actually indicate the described threat. Keep in mind this number is always set by the provider of the indicator. The usefulness of this number is primarily for security investigators, as they can leverage this value to influence their urgency to respond to the threat. One could also author analytics rules that used this value to make determinations on alert severity, aggregation behaviors, etc. depending on higher or lower confidence values.
Q: Do Watchlist support multiple workspaces?
A Watchlist can be used in queries only within the current workspace. You would need to create a copy of the Watchlist in each workspace, or use an alternative lookup method as described here.
Module 8: Analytics
Q: Are there any restrictions to queries used in Azure Sentinel rules?
Azure Sentinel supports Log Analytics KQL queries; those may somewhat differ from Azure Data Explorer KQL queries.
Also, queries used in alert rules have the following limitations:
- The query max length is 10000
- Cannot contain "search *" and "union *".
Q: How many analytic rules can I define?
512 per workspace
Q: The field I need is not available for entity mapping. Why?
If the field you want to map to an entity in the alert rule configuration screen is not available, the chances are that the value is not a string.
You can check that by trying to manually map as part of the query by adding to the query an "extend" operation:
| extend AccountCustomEntity = your_value
If you get the error "Entity mapping conflict - make sure you choose the correct property type," the issue is indeed the value type.
To solve typecast to string the value using the "tostring" function:
To solve typecast to string the value using the "tostring" function:
| extend AccountCustomEntity = tostring(your_value)
Q: My query works in the log screen but not when creating an alert rule?
This usually implies that the KQL query relies on specific data to work. A common example of this is using bag_unpack, which generates fields based on the data; As the source data changes, those fields may not always be available, leading to the rule execution failure. Therefore the KQL validator rejects the rule KQL.
The correct way to use fields generated by bag_unpack is using the column_ifexists function:
The correct way to use fields generated by bag_unpack is using the column_ifexists function:
SecurityAlert
| extend custom_details_temp = parse_json(ExtendedProperties)
| evaluate bag_unpack (custom_details_temp, "custom_")
| project custom_IncidentId = column_ifexists("custom_IncidentId", "")
Q: Is there a way to get a list of the built-in rule templates?
Use the PowerShell script here. Note that it enumerates the rules in GitHub, and it might take a couple of weeks for new rules to be available in the gallery. You can configure them yourself in the meanwhile.
Module 10: Workbooks, reporting, and visualization
Q: Can I add custom Images to a workbook?
You can insert images in a markdown (text) steps in a workbook using the markdown image syntax. The text's content can also use workbook parameters if you want the paths to change based on parameter values.
Q: Can I embed videos in a workbook?
Not at this time, though animated images will work.
Module 12: A day in a SOC analyst's life, incident management, and investigation
Q: How do I get a notification when a resource is updated?
- When rule templates are updated, the template is flagged as "new" in the UI.
- When a workbook is updated, you are notified in the UI to update it.
- For other resources subscribing to notifications on GitHub
Q: How are incidents updates when Microsoft alerts are updated?
When using Microsoft rules which create incidents directly from an alert from Microsoft products, Azure Sentinel handles updates for those alerts automatically:
- For a new alert arrives, a new incident is created. If the alert is sent as resolved, the incident will be created as resolved.
- If an incident for the alert (meaning, SystemAlertId) already exists, Azure Sentinel updates the incident but will not change its status.
- However, when presented with an alert, Azure Sentinel looks only 1 month back for existing incidents. This means that if an alert is resolved at the providers' after, say, 50 days, a new resolved incident will be created for that alert update.
Q: Any limit on number of comments for an incident?
Yes. You can add up to a 100 comments to an incident.
Module 13: Hunting
Q: Is there a reason to choose the MITRE attacks tactic in Sentinel for Hunting?
A hunting campaign has to start with a strategy – where do I hunt? This translates to filtering the hunting queries in Azure Sentinel and running the relevant queries to your starting point. A strategy that takes a specific MITRE tactic as a starting point is a popular one.
7 Comments
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.