In this blog post I am going to talk about splitting logs to multiple tables and opting for basic tier to save cost in Microsoft Sentinel. Before we delve into the details, let’s try to understand what problem we are going to solve with this approach.
Azure Monitor offers several log plans which our customers can opt for depending on their use cases. These log plans include:
Following diagram provides detailed information about the log plans and their use cases:
I would also recommend going through our public documentation for detailed insights about feature-wise comparison for the log plans which should help you in taking right decisions for choosing the correct log plans.
**Note** Auxiliary logs are out of scope for this blog post, I will write a separate blog on the Auxiliary logs later.
So far, we know about different log plans available and their use cases.
The next question is which tables support Analytics and Basic log plan?
You can switch between the Analytics and Basic plans; the change takes effect on existing data in the table immediately.
When you change a table's plan from Analytics to Basic, Azure monitor treats any data that's older than 30 days as long-term retention data based on the total retention period set for the table. In other words, the total retention period of the table remains unchanged, unless you explicitly modify the long-term retention period.
Check our public documentation for more information on setting the table plan.
I will focus on splitting Syslog table and setting up the DCR-based table to Basic tier in this blog.
Typically Firewall logs contribute to high volume of log ingestion to a SIEM solution.
In order to manage cost in Microsoft Sentinel its highly recommended to thoroughly review the logs and identify which logs can be moved to Basic log plan.
At a high level, the following steps should be enough to achieve this task:
At this point, I anticipate you already have log forwarder set up and able to ingest Firewall logs to Microsoft Sentinel’s workspace.
Let’s focus on creating a custom table now
This part used to be cumbersome but not anymore, thanks to my colleague Marko Lauren who has done a fantastic job in creating this PowerShell Script which can create a custom table easily. All you need to do is to enter the pre-existing table name and the script will create a new DCR-Based custom table with same schema.
Let’s see it in action:
This should create a new DCR-based custom table which you can check in Log Analytics Workspace > Table blade as shown below:
**Note** We highly recommend you should review the PowerShell script thoroughly and do proper testing before executing it in production. We don't take any responsibility for the script.
The next step is to update the Data Collection Rule template to split the logs
Since we already created custom table, we should create a transformation logic to split the logs and send less relevant log to the custom table which we are going to set to Basic log tier.
For demo purposes, I’m going to split logs based on SeverityLevel. I will drop “info” logs from Syslog table and stream it to Syslog_CL table.
Let’s see how it works:
Let’s validate if it really works
Go to the Log Analytics Workspace > Logs and check if the tables contains the data which we have defined it for.
In my case as we can see, Syslog table contains all logs except those where SeverityLevel is “info”
Additionally, our custom table: Syslog_CL contains those Syslog data where SeverityLevel is “info”
Now the next part is to set the Syslog_CL table to Basic log plan
Since Syslog_CL is a DCR-based custom table, we can set it to Basic log plan. Steps are straightforward:
Now you can enjoy some cost benefits, hope this helps.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.