(Updated April 25th 2021)
While extensive, the Ninja training has to follow a script and cannot expand on every topic. Like any training, you may have questions after the session. This live blog post tries to address that by providing answers to common questions ordered by the Ninja training modules.
Let go!
Module 1: Get started with Azure Sentinel
Q: How do I do a free-of-charge trial for Azure Sentinel?
There is no straight forward free trial for Sentinel:
- Every new workspace is not billed for *Azure Sentinel* for a month.
- However, the Azure Sentinel cost is made of the Azure Sentinel cost and the Log Analytics cost, and there is *no free trial for Log Analytics*.
There is, however, some usage that is always free, and you try to limit yourself to those to have a free POC:
- Log Analytics is free for the first 5GB for each month, across an *account*
- Both Log Analytics and Sentinel are free when Sentinel is deployed for selected sources such as Office 365.
So, how do I run a free PoC? Either of those:
- Using free sources only.
- On top of an existing, already paid for Log Analytics data. Giving 30 days of free Sentinel ingestion.
- A dedicated Azure tenant unrelated to the EA gives 30 days of free Sentinel ingestion and 5GB/m free Log Analytics ingestion. The 30 days can be restarted by creating a new workspace.
Q: Is there a certification for Azure Sentinel? For the Ninja Training?
The new SC-200 exam (Microsoft Security Operations Analyst) covers also Azure Sentinel, which is 40% to 45% of the exam, alongside Microsoft Defender and Azure Defender, which are great complements. The SC-200 is not a Ninja Training certification, but the exam is largely based on Ninja Training materials, making it a good learning path for the certification.
Q: How can I send sample data?
For CEF (CommonEventLog) events stored in a file, you can use Logstash to read data from your CEF sample log file and send it directly into the Log Forwarder.
{
input {
file {
path => "/home/stefan/samplelogs/cef.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
output {
# change to your log forwarder host and port
tcp {
host => "127.0.0.1"
port => 514
}
}
Q: How can I have a direct link to the Azure Sentinel overview page? Any other page?
You don't need to get to Azure Sentinel through the Azure Portal every time. Just bookmark any page (or copy the URL) and use it to access your favorite starting point. The URL will have the following format, with the blade number changing based on the specific page you wanted to start with (line breaks added for calrity):
https://portal.azure.com/#blade/Microsoft_Azure_Security_Insights/MainMenuBlade/11/
subscriptionId/<your-subscription-id>/
resourceGroup/<your-resource-group-name>/
workspacename/<your-resource-workspace-name>
Pricing and billing
The Azure Sentinel Usage Workbook provides the most comprehensive information on use. For actual billing information use the Azure portal cost management screen. Filter by the scope relevant to you (the workspace or resource group, for example).
The usage information is available in the workspace, and you can use these queries to report or as a starting point for your reporting. The usage reporting workbooks for Azure Sentinel uses this information to provide a comprehensive view of usage.
Q: I used the pricing calculator, is this the actual cost I will pay?
The pricing calculator is a starting point. The following might imply your cost is actually lower:
- Different Azure Regions have very different prices. Technically, if there is no regulatory pressure, there is no reason not to pick the cheapest region (US East or US West 2 at the time of writing):
- Neither collection latency, nor user interface latency is of any significance. We have customers in Japan using US data centers without any issue. You can find infomation about Azure cross-region latency here.
- When collecting from Azure regions, there is some cross-region networking charge, however as noted later in this document, it does not cover all communication and in any case much lower than the difference between regions.
- There is a specific cost-benefit to using both Azure Defender and Sentinel. Each Azure Defender license entitles the customer to 500 MB/d free Windows Security collection on the Log analytics part of the Sentinel cost, which often amounts to a large reduction in cost. To enable this free consumption, the Windows Security events should be collected to the Sentinel workspace, which is the current best practice.
- Your current enterprise agreement with Microsoft might already include a discount, either a direct percentage discount or in the form of an Azure commitment, which will apply to Sentinel.
Our official pricing is to charge for retention beyond 90 days for sources ingested for free. However, you may find that in some cases, we do not actually charge. While we may start charging for such retention in the future, we will not charge for past charges not collected.
Network communication between regions in Azure costs money, and the question is, how does this relate to Azure Sentinel?
Telemetry collected using an agent, the Log forwarder, or custom connectors using the ingest API, if the relevant source is not in the workspace region, would incur inter-region bandwidth costs.
However, service-to-service connectors, including Azure diagnostics sources, Office 365, and Microsoft 365 sources will not incur such costs even if the telemetry source is in a different region than the workspace. For example, if you collect telemetry from an Azure Firewall, there is no bandwidth charge regardless of the firewall region.
Please note that the Azure Sentinel documentation is incorrect and identity several agent-based sources such as DNS and Windows Firewall as service to service connectors.
Q: When I enable Azure Sentinel on an existing Log Analytics workspace, how does pricing change?
- There will be an additional cost for Sentinel applied to all data in the workspace.
- All data will be retained for 90 days with no additional charge. Additional retention remains at the current Log Analytics rate.
- The sources free for Sentinel ingestion will not be charged for ingestion (Log Analytics or Sentinel tier). Retaining this data beyond 90 days costs at the Log Analytics retention price.
Q: Can Azure Sentinel capacity reservations be reserved for 1 year, 3 years?
No. Azure Sentinel capacity reservations are different from Azure reserved instances and behave like standard Azure meters, billed daily. They differ from pay-as-you-go pricing as they offer a lower per-unit price for reserving a larger amount of units.
Q: How does using Azure Defender affects Azure Sentinel pricing?
When collecting information from Azure Defender licensed nodes into an Azure Sentinel enabled workspace, 500MB/d per licensed node are deducted from the cost of certain log types from the Log Analytics price for the workspace, but not from the Azure Sentinel price. The list of relevant log types can be found here. Additional information on this allowance can be found in the Azure Defender pricing FAQ.
Q: Why is the pricing calculator using different capacity reservations for Log Analytics and Azure Sentinel?
Regulation and Compliance
Q: Does Azure Sentinel store all data locally?
While the official Azure data residency page mentions that Azure Sentinel is an exception and does not store all data within geography, Azure Sentinel does store data locally in a (growing) number of geographies as outlined here.
Module 2: How is Azure Sentinel used?
Azure Sentinel as part of the Microsoft Security stack
Q: On a Windows system with Defender for endpoints already installed, would you install the Log Analytics agent to report Security Events to Azure Sentinel as well?
In general, the answer is yes, but it would depend on the use cases. Windows events are wide in scope but broadly fall into two groups:
- Activity (such as process, file, and network activity) that overlap with MDATP.
- Management audit (for example, user management) is not in the MDATP domain.
Q: How does Azure Sentinel compare with the Graph Security API?
The main difference is that the Graph Security API does not support raw telemetry, which is the bread and butter of Azure Sentinel. The Sentinel connectors focus on getting raw telemetry. There are exceptions in areas we need to improve the cross-utilization, and we are working on that.
Side by side with your existing SIEM
Q: How do I forward alerts from Azure Sentinel to another system?
See the Ninja training side-by-side section.
Q: How do I forward data, alerts, or events from my current SIEM to Azure Sentinel?
The most common way would be to use Syslog or CEF, which most SIEM products support. Note that you would like to forward from the 3rd party SIEM collector layer in many cases, which is more efficient than overloading the 3rd party SIEM processing layer.
The following links can get you started:
- Splunk:
- ArcSight (CEF),
- Kiwi Syslog Server (Syslog),
- SolarWinds (Syslog)
Q: Ticket System Integration? Is it ServiceNow only?
While ServiceNow is the most popular ticketing system and many of our examples are focused on it, Logic Apps, on which the integration is based, has connectors with other ticketing systems:
If not available, you can still connect to your ticketing systems using a custom Logic App connector, the HTTP connector that supports most APIs, or an Azure function from Logic Apps.
Q: How do I forward events from Azure Sentinel to another SIEM?
We do not recommend forwarding all events from Azure Sentinel to your on-prem SIEM. It may imply you are not getting enough value from Azure Sentinel and worth looking into.
In case you want to forward events (all of some), export from Azure Sentinel / Log Analytics to Azure Storage and Event Hub or move Logs to Long-Term Storage using Logic Apps.
In Azure Gov (the US federal clouds)
Q: Is Azure Sentinel FedRAMP and DoD CC SRG certified?
Yes. See here.
Q: Are there known limitations to Azure Sentinel in Azure Gov?
You can find the known limitations of Azure Gov here.
Module 3: Workspace and tenant architecture
Q: Best practice is to minimize the number of workspaces, but I want to split the bill. How do I do that?
Read how to report on the ingestion volume per computer, resource, resource group or subscription.
Q: Are the best practices for Log Analytics and Azure Sentinel concerning workspace architecture the same?
Not always. Log Analytics and Azure Sentinel have different use cases and users, which sometimes require a different approach. If Azure Sentinel uses a workspace, use the Azure Sentinel best practices. Also, try to minimize the amount of data not relevant to Azure Sentinel in the workspace to avoid unnecessary costs.
As a reference, you can find the Log Analytics multi-workspace best practices here:
Q: Can I move the Azure Sentinel workspace to a different Resource Group of subscription?
While the feature is available for a Log Analytics workspace, we have not comprehensively tested moving an Azure Enabled workspace to a new subscription. Customers have done it before, and the one issue we encountered was that analytics rules do not work anymore, disabling and enabling the rules help. That said, there might be other issues, so the prudent solution would be to start over.
Module 4: Collecting events
General
Log Forwarder
Note that the Log Forwarder is based on the Linux based Log Analytics Agent (MMA), so the questions in the next section, as far as they pertain to the Linux MMA, are relevant for the Log Forwarder as well.
Q: How do I set the Log Forwarder to listen to encrypted Syslog
Configure the Syslog server part of the Log Forwarder (rsyslog or Syslog-NG) to listen to TLS based Syslog:
Q: Can I filter Syslog of CEF events?
Yes, See the Log Forwarder webinar: YouTube, MP4, Deck.
Q: Should I filter firewall events?
Unlike windows events, Firewall events are simple and of only a handful of types. The most common event types (using Palo Alto's terminology) are:
- Traffic events - any connection through the Firewall.
- Threat events - any URL accessed through the Firewall (the name is misleading here)
Both have significant value for your security but have a large volume and therefore cost. Preferably, all should be collected. Inbound failures are candidates for filtering out, as they include a huge volume of low quality attack attempts.
Q: What size VM should I use for the Log Forwarder?
The Log Forwarder does little itself as parsing is done in the cloud. Therefore, comparatively, smaller and cheaper systems can be used.
You can find official sizing information in the documentation.
In addition, recent reports from customers have suggested:
- 500 GB/d of CEF data using a three VM scale set of Standard_D4s_v3 (4 CPU, 16GB) VMs.
- 6000 EPS of CEF data using a single physical VM: 8 vCPUs, 16 GB memory, Intel Xeon Platinum 8171M CPU @ 2.60GHz.
Use a VM scale set with an Azure load balancer or an on-prem load balanced to go beyond.
Q: Does the Log Forwarder cache information in case of a network outage?
Yes. See details here.
Log Analytics Agent and Azure Monitor Agent
Q: Is the workspace key stored on the agent machine?
We don't store the workspace key. It's only used during onboarding to generate the certs used for on-going communications by the Agent. The Workspace ID is stored in a config file per workspace here: /etc/opt/microsoft/omsanget/ws-id.
Q: Can Azure Sentinel filter Windows Events?
The Log Analytics agent (MMA) offers limited control over the Windows events forwarded. You can set a collection tier for all agents. However, the common tier is often not enough for Azure Sentinel customers, especially as it has to be set for all agents.
The new Azure Monitoring Agent (AMA) can granularly filter Windows events using WEF like XPath expressions.
Q: Does the Agent compress data from on-prem to the cloud?
Yes, the Log Analytics agent (MMA) compresses data when sending it to the cloud. This is used for Syslog, CEF, and local Windows or Linux telemetry. For Linux, the agent uses Zlib compression. The lib compression ratio is typically between 2:1 to 5:1 and maxes out theoretically at 1032:1
Q: Are there limits to how many custom logs (i.e. files) the Log Analytics agent can collect
The Log Analytics agent can collect files located on the machine it is installed on. This feature is intended for collecting local files and not as a means for aggregated collection, for example replacing Syslog. It is therefore limited to 500 EPS (Events, or log lines, per second) and exhibits issues if attempting to collect and forwards higher rates. A common issue that happens at higher rates is event duplication. If you need to collect files at a high volume into Azure Sentinel, consider using Logstash as described here.
Q: Does the Log Analytics agent cache information in case of a network outage?
Yes. For the Linux agent see details here.
Specific connectors
Q: Can I connect two workspaces using the Microsoft 365 defender connector? If so, does incident synchronization behave?
You can connect two workspaces to Microsoft 365 defender (M365D), and incidents will be synchronized between both workspaces and M365D. In practice, when you change status in one workspace, M365D will be updated, and on the next sync cycle (which occurs every 5 minutes), the other workspace will pick the changes.
Q: The Microsoft Defender for Office (Office ATP) connector does not collect all alerts. What can I do?
The Microsoft Defender for Office alerts connector collects only AIRS alerts. To collect other alerts use the Office 365 custom connector.
Specifically, you will find the relevant alerts under those record types:
- 28 (ThreatIntelligence) - Phishing and malware alerts from Exchange Online Protection and Office 365 Advanced Threat Protection.
- 41 (ThreatIntelligenceUrl) - ATP Safe Links time-of-block and block override alerts from Office 365 Advanced Threat Protection.
- 47 (ThreatIntelligenceAtpContent) - Phishing and malware alerts for files in SharePoint Online, OneDrive for Business, and Microsoft Teams from Office 365 Advanced Threat Protection.
- 64 (AirInvestigation) - Automated investigation and response alerts, such as investigation details and relevant artifacts from Office 365 Advanced Threat Protection Plan 2.
Start your queries with the following snipper to get alerts of a specific type, substituting 28 for the other types above:
FAOfficeActivityALL_CL | where RecordType_d == "28"
Q: Does the Azure Information Protection connector support AIP Unified Labeling?
Yes. The same connector collects both AIP Classic and AIP Unified Labeling (UL) logs.
Q: The Teams connector does not support Teams Shifts audit. How can I collect it?
To collect Team Shifts alerts use the Office 365 custom connector and query for RecordType_d == "73".
Q: Which API does Azure Sentinel to collect CloudTrail events
The AWS CLoudTrail API LookupEvents end point.
- Start an Administrative Command Prompt and run 'Net Stop HealthService'
- Start File Explorer and navigate to C:\Program Files or C:\Program Files(x86)
- Go to this location: Microsoft Monitoring Agent\Agent
- Rename the folder Health Service State to Old Health Service State
- In the Administrative Command Prompt, run Net Start Health Service
input(type="imtcp" port="<TCP_PORT>" supportOctetCountedFraming="off")Module 5: Log Management
Q: How can I learn about the schema of the tables in Azure Sentinel?
In general, schema references can be found in the reference section of the Azure Sentinel docs. Those are a few resources to start with:
- Data collected from external sources:
- Internal tables:
Q: The log search is limited to 30K results; what can I do?
Indeed, there is a 30K cap on the result set size in the UI. There is usually not meaningful need to review so many results in the UI. The API, and hence PowerShell, can return up to 500,000 results. Use the PowerShell script to run a query and get the results in a CSV file.
If you still need more than 10K results in the portal:
- You can transform your results into an array, which can hold much more than 10K values.
- Reduce the size of your results - you can use "distinct Computer," "summarize by Computer," or "summarize make_set" to remove duplicate values from your results (Also, if all you need is that computer's name, "project" only that column)
Q: Which columns are displayed in a search result if not specifically projected?
Multiple heuristics determine which fields to display. Some common ones are:
- Hiding system columns that typically pollute the visual space and are not commonly used (_ResourceId, for example)
- Hiding any columns that do not contain any data for the entire result set
-
Hiding by default predefined columns for specific tables.
Q: Can I delete unused custom log tables from a workspace?
The tables will disappear once empty. Use the purge API or wait for the retention period to end.
Standard fields include event time fields, record type, and billing information fields. See Standard properties in Azure Monitor Logs for more details.
Q: we can guarantee the data that has been ingested into Azure Sentinel cannot be tampered with?
Data in database storage cannot be altered once ingested but can be deleted using the purge API. Although data cannot be altered, some certifications require that data is kept immutable and cannot be changed or deleted in storage. Data immutability can be achieved using data export to a storage account that is configured as immutable storage.
Module 6: Enrichment: TI, Watchlists, and more
Q: How often does Azure Sentinel Poll TAXII for new IOCs, and can this be configured?
This depends on the TAXII server. Generally speaking, if a well-formed TAXII server adheres to the standards, the TAXII data connector will pull the entire collection on the first connection and then pull only incremental changes every minute.
Q: What information from the TAXII server does Azure Sentinel pull
Currently, Azure Sentinel requests from the TAXII server and ingests only indicator STIX objects. We are planning the support of other STIX Domain Objects in the future. We perform a mapping from STIX to the ThreatIntelligenceIndicator table schema when we import the data.
Q: Is pagination supported in TAXII?
Yes, we support pagination. The TAXII server determines the size of the page. The TAXII server that you are connected to decides the number of IOC's to be returned in a request.
Q: Do we have specific IP addresses that we would use to pull this data into Sentinel?
While there are no specific IP addresses, they will be Azure IP addresses within the relevant workspace region. You can find the list of Azure IP addresses here (the list is dynamic).
Q: Since the Graph Security API is a tenant level, can one control what threat indicators each workspace receives?
Q: How do I use the confidence score associated with threat intelligence IoCs?
The confidence score is meant to convey the level of certainty the provider of the threat indicator feels the observations of the pattern in the indicator actually indicate the described threat. Keep in mind this number is always set by the provider of the indicator. The usefulness of this number is primarily for security investigators, as they can leverage this value to influence their urgency to respond to the threat. One could also author analytics rules that used this value to make determinations on alert severity, aggregation behaviors, etc., depending on higher or lower confidence values.
Q: Events in the CommonSecurityLog (CEF) tables include threat intelligence information. Where does it come from?
An internal process matches IP addresses from CEF logs to an internal Microsoft threat intelligence platform and extends rows with additional information when matches are found. If the customer believes this to be a false positive, they should open a support ticket.
Q: Do Watchlist support multiple workspaces?
A Watchlist can be used in queries only within the current workspace. You would need to create a copy of the Watchlist in each workspace or use an alternative lookup method as described here.
Module 8: Analytics
Q: Are there any restrictions to queries used in Azure Sentinel rules?
Azure Sentinel supports Log Analytics KQL queries; those may somewhat differ from Azure Data Explorer KQL queries.
Also, queries used in alert rules have the following limitations:
- The query max length is 10000
- Cannot contain "search *" and "union *".
Q: The field I need is not available for entity mapping. Why?
If the field you want to map to an entity in the alert rule configuration screen is not available, the chances are that the value is not a string.
You can check that by trying to manually map as part of the query by adding to the query an "extend" operation:
| extend AccountCustomEntity = your_value
To solve typecast to string the value using the "tostring" function:
| extend AccountCustomEntity = tostring(your_value)
The correct way to use fields generated by bag_unpack is using the column_ifexists function:
SecurityAlert
| extend custom_details_temp = parse_json(ExtendedProperties)
| evaluate bag_unpack (custom_details_temp, "custom_")
| project custom_IncidentId = column_ifexists("custom_IncidentId", "")
Q: How can I learn about the schema of the tables in Azure Sentinel?
See the Log Management module FAQ above.
Module 9: SOAR
Q: Does Azure Sentinel support on-prem automation?
Yes. If the target on-prem system supports a rest API, the Logic Apps on-prem gateway can be used. To run any command on-prem, use Azure Automatin in conjuction with Logic Apps as described in the blog post: Automatically disable On-prem AD User using a Playbook triggered in Azure.
Module 10: Workbooks, reporting, and visualization
Q: Can I add custom Images to a workbook?
You can insert images in a markdown (text) steps in a workbook using the markdown image syntax. The text's content can also use workbook parameters if you want the paths to change based on parameter values.
Q: Can I embed videos in a workbook?
Not at this time, though animated images will work.
Module 12: A day in a SOC analyst's life, incident management, and investigation
Q: How do I get a notification when a resource is updated?
- When rule templates are updated, the template is flagged as "new" in the UI.
- When a workbook is updated, you are notified in the UI to update it.
- For other resources subscribing to notifications on GitHub
Q: How are incidents updates when Microsoft alerts are updated?
When using Microsoft rules which create incidents directly from an alert from Microsoft products, Azure Sentinel handles updates for those alerts automatically:
- For a new alert arrives, a new incident is created. If the alert is sent as resolved, the incident will be created as resolved.
- If an incident for the alert (meaning, SystemAlertId) already exists, Azure Sentinel updates the incident but will not change its status.
- However, when presented with an alert, Azure Sentinel looks only 1 month back for existing incidents. This means that if an alert is resolved at the providers' after, say, 50 days, a new resolved incident will be created for that alert update.
Q: Any limit on number of comments for an incident?
Yes. You can add up to a 100 comments to an incident.
Q: It is not enough to block an Office 365 user when a breach is detected. How do I kill active sessions?
See here.
Module 13: Hunting
Q: Is there a reason to choose the MITRE attacks tactic in Sentinel for Hunting?
A hunting campaign has to start with a strategy – where do I hunt? This translates to filtering the hunting queries in Azure Sentinel and running the relevant queries to your starting point. A strategy that takes a specific MITRE tactic as a starting point is a popular one.
Module 15: Monitoring Azure Sentinel's health
Q: How do I learn about service distruptions?
The Azure Services Status page should be your first place to look at. This page enables identifying issues in all the services supporting Azure Sentinel, including Log Anlaytics, Logic Apps and Azure Sentinel itself.
For more details on issues in Azure Monitor, refer to the Azure Monitor refer to the Azure Monitor Status blog.
Microsoft
