Note: Thank you to @Yaniv Shasha , @Sreedhar_Ande , @JulianGonzalez , and @Ben Nick for helping deliver this preview.
We are excited to announce a new suite of features entering into public preview for Microsoft Sentinel. This suite of features will contain:
Basic Ingestion Tier:
The basic log ingestion tier will allow users to pick and choose which data tables should be enrolled in the tier and ingest data for less cost. This tier is meant for data sources that are high in volume, low in priority, and are required for ingestion. Rather than pay full price for these logs, they can be configured for basic ingestion pricing and move to archive after the 8 days. As mentioned above, the data ingested will only be retained in the workspace for 8 days and will support basic KQL queries. The following will be supported at launch:
Note: this data will not be available for analytic rules or log alerts.
During public preview, basic logs will support the following log types:
Note: More sources will be supported over time.
Archive Tier:
The archive tier will allow users to configure individual tables to be retained for up to 7 years. This introduces a few new retention policies to keep track of:
Data tables that are configured for archival will automatically roll over into the archive tier after they expire from the workspace. Additionally, if data is configured for archival and the workspace retention (say 180 days) is lowered (say 90 days), the data between the original and new retention settings will automatically be rolled over into archive in order to avoid data loss.
Configuring Basic and Archive Tiers:
In order to configure tables to be in the basic ingestion tier, the table must be supported and configured for data collection rules for custom logs. For steps to configure this, please follow this document. Archive does not require this but it is still recommended.
Currently there are 3 ways to configure tables for basic and archive:
REST API
The API supports GET, PUT and PATCH methods. It is recommended to use PUT when configuring a table for the first time. PATCH can be used after that. The URI for the call is:
https://management.azure.com/subscriptions/<subscriptionId>/resourcegroups/<resourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<workspaceName>/tables/<tableName>?api-version=2021-12-01-preview
This URI works for both basic and archive. The main difference will be the body of the request:
Analytics tier to Basic tier
{
"properties": {
"plan": "Basic"
}
}
Basic tier to Analytics tier
{
"properties": {
"plan": "Analytics"
}
}
Archive
{
"properties": {
"retentionInDays": null,
"totalRetentionInDays": 730
}
}
Note: null is used when telling the API to not change the current retention setting on the workspace.
PowerShell
A PowerShell script was developed to allow users to monitor and configure multiple tables at once for both basic ingestion and archive.
The script and workbook can be found here and here.
To configure tables with the script, a user just needs to:
Workbook
A workbook has been created that can be deployed to a Microsoft Sentinel environment. This workbook allows for users to view current configurations and configure individual tables for basic ingestion and archive. The workbook uses the same REST API calls as listed above but does not require authentication tokens as it will use the permissions of the current user. The user must have write permissions on the Microsoft Sentinel workspace.
To configure tables with the workbook, a user needs to:
The workbook will run the API call and will provide a message if it was successful or not. The changes made can be seen after refreshing the workbook.
Both the PowerShell script and the Workbook can be found in the Microsoft Sentinel GitHub repository.
Search Jobs:
Search jobs allow users to specify a data table, a time period, and a key item to search for the in the data. As of now, Search jobs use simple KQL, which will support more complex KQL over time. In terms of what separates Search jobs from regular queries, as of today a standard query using KQL will return a maximum of 30,000 results and will time out at 10 minutes of running. For users with large amounts of data, this can be an issue. This is where search jobs come into play. Search jobs run independently from usual queries, allowing them to return up to 1,000,000 results and up to 24 hours. When a Search job is completed, the results found are placed in a temporary table. This allows users to go back to reference the data without losing it and being able to transform the data as needed.
Search jobs will run on data that is within the analytics tier, basic tier, and also archive. This makes it a great option for bringing up historical data in a pinch when needed. An example of this would be in the event of a widespread compromise that has been found that stems back over 3 months. With Search, users are able to run a query on any IoC found in the compromise in order to see if they have been hit. Another example would be if a machine is compromised and is a common player in several raised incidents. Search will allow users to bring up historical data from the past int the event that the attack initially took place outside of the workspace's retention.
When results are brought in, the table name will be structured as so:
Example: SecurityEvents_12345_SRCH
Data Restoration:
Similar to Search, data restoration allows users to pick a table and a time period in order to move data out of archive and back into the analytics tier for a period of time. This allows users to retrieve a bulk of data instead of just results for a single item. This can be useful during an investigation where a compromise took place months ago that contains multiple entities and a user would like to bring relevant events from the incident time back for the investigation. The user would be able to check all involved entities by bringing back the bulk of the data vs. running a search job on each entity within the incident.
When results are brought in, the results are placed into a temporary table similar to how Search does it. The table will take a similar naming scheme as well:
Example: SecurityEvent_12345_RST
Performing a Search and Restoration Job:
Search
Users can perform Seach jobs by doing the following:
Once this has been performed, a new Search job will be created. The user can leave and come back without impacting the progress of the job. Once it is done, it will show up under saved searches for future reference.
Note: Currently Search will use the following KQL to perform the Search: TableName | where * has 'KEY TERM ENTERED'
Restore
Restore is a similar process to Search. To perform a restoration job, users need to do the following:
Pricing
Pricing details can be found here:
While Search, Basic, Archive, and Restore are in public preview, there will not be any cost generated. This means that users can begin using these features today without the worry of cost. As listed on the Azure Monitor pricing document, billing will not begin until April 1st, 2022.
Search
Search will generate cost only when Search jobs are performed. The cost will be generated per GB scanned (data within the workspace retention does not add to the amount of GB scanned). Currently the price will be $0.005 per GB scanned.
Restore
Restore will generate cost only when a Restore job is performed. The cost will be generated per GB restored/per day that the table is kept within the workspace. Currently the cost will be $.10 per GB restored per day that it is active. To avoid the recurring cost, remove Restore tables once they are no longer needed.
Basic
Basic log ingestion will work similar to how the current model works. It will generate cost per GB ingested into Azure Log Analytics and also Microsoft Sentinel if it is on the workspace. The new billing addition for basic log ingestion will be a query charge for GB scanned for the query. Data ingested into the Basic tier will not count towards commitment tiers. Currently the price will be $.50 per GB ingested in Azure Log Analytics and $.50 per GB ingested into Microsoft Sentinel.
Archive
Archive will generate a cost per GB/month stored. Currently the price will be $.02 per GB per month.
Learn More:
Documentation is now available for each of these features. Please refer to the links below:
Additionally, helpful documents can be found in the portal by going to 'Guides and Feedback'.
Manage and transform your data with this suite of new features today!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.