May 04 2020 07:48 AM - edited May 04 2020 07:57 AM
We have been using Sentinel in conjunction with Azure Log Analytics for quite some time to ingest selected security logs (AD, DNS, Windows Security etc.) from VM-agents in our server environment. Last week we upgraded the workspace to enable the newly released "Azure Monitor for VMs" and also installed the Service Map agents on our VMS. This resulted in a huge increase of data ingested into Sentinel (5x, see attached image) due to VM performance metrics and network traffic logs from the the Service Map now being ingested into Sentinel, that I have no interest in having there.
How can I choose/filter what logs from a Log Analytics Workspace that is forwarded to Azure Sentinel?
May 04 2020 10:07 AM - edited May 04 2020 10:10 AM
@Magnus Tjerneld You cannot state which information stored in a Log Analytics workspace is available to Azure Sentinel. It is an all or nothing proposition.
You can control what data gets ingested from the VMs by clicking on the "Windows, Linux, and other sources" link on the Overview page and from there go to the "Data" section. This will show you what logs you are ingesting as well as severity for each log. You can go through the list and see if you can trim down some of the information being imported.
May 04 2020 11:19 AM
Ok; that's too bad. I dug a little deeper into the increased log volumes and realized that the bulk of the records/sources are not visible under Data Connections.
Previously; before we upgraded the Workspace to onboard "Azure Monitor for VM:s" I was manually pulling a number of selected perfmon counters (CPU, RAM and so on). I did this at a 5 min interval to give me an overview without causing too much data in OMS. But after enabling Azure Monitor for VM:s, a huge number of new counters has been enabled and the resolution is much higher. Over the last 24h I've recieved ~300k datapoints for the 4 VMs I'm monitoring:
Computer | 11,499 | ||
LogicalDisk | 205,315 | ||
Memory | 5,744 | ||
Processor | 5,744 | ||
Network | 14,360 |
None of these namespaces are visible under Data Sources in Log Analytics.
It would be great to be able to lower the resolution/interval, but I can't seem to find anywhere to limit the interval on these logs?
I realize that this is now more of an "Azure Monitor for VM:s" question than a Sentinel question.
May 04 2020 01:25 PM
@Magnus Tjerneld Did you check under the "Windows Performance Counters" in the "Data" section?
May 04 2020 11:35 PM
@Gary Bushey Yes; only the manual counters that I had set up before I enabled "Azure monitor for VMs" are visible there. AMFVM seems to set up it's own data collection that you don't seem to be able to edit.
May 05 2020 12:55 AM
Please see the GA release info
https://docs.microsoft.com/en-us/azure/azure-monitor/insights/vminsights-ga-release-faq
May 05 2020 01:22 AM - edited May 05 2020 01:22 AM
Thanks @CliveWatson. I read this before and have now read it again; and I realize that I can delete my old perfmon counters. However, I do not find any information regarding:
- Can I limit the "resolution" of data performance data sent to Log Analytics after upgrading to Azure Monitor for VMs? In the old solution, I could set intervals in seconds.
- Can I choose not to collect data for a specific namespace? For us, Disk-metrics make up 90% of logs ingested and causes a lot of extra costs in Sentinel. If possible, I'd opt out.
And my wish would be to be able to exclude all performance counters from ingestion into Sentinel. This only results in added cost and no added value.
Sep 24 2020 09:16 AM
@Magnus Tjerneld Were you able to find a solution to this? Filtering out which data that is ingested by sentinel?
Sep 24 2020 01:51 PM