Oct 10 2017 05:08 AM
Oct 10 2017 05:08 AM
I'm trying to use OMS alerts for heartbeat of my servers. On the old Log analytics, i readed an article saying thats the agent reports each 60 seconds for the OMS and the results of the queries is normally, obeying the frequency of agent reports.
After migration to the new Log Analytics, my query return a great delay on the registers of the heartbeats, having 30 minutes of late for the registers answered.
Anyone could clarify if we have some changes or trouble on the agents or the results of queries on the new Log Analytics?
Oct 10 2017 06:23 AM
I've noticed a little bit of a larger delay in event and heartbeat data being recorded in OMS - in some testing I saw heartbeat data taking up to 8 minutes to show up in log searches, where it had previously usually taken at most 3 minutes. The heartbeat data is still recorded every 60 seconds - but the indexing and displaying in the log search can take longer.
Oct 11 2017 03:59 AM
On this test i have about ~20 minutes in a simple query results.
Oct 11 2017 11:07 PMSolution
If you are quering this way
You will get inconsistant alerts. With the new query langauge you have to specify the sorting. In the old one sorting was always done on TimeGenerated if not specified. So you have to do query like this to get the latest data
Heartbeat | sort by TimeGenerated desc
Otherwise there is no change on how logs are ingested into Log Analytics.
Nov 03 2017 07:13 AM
I've noticed the same problem in my workspace. It doesn’t matter if you sort your query results by TimeGenerated or not. The Heartbeats are just late in some cases... Sometimes up to 20 minutes, which makes alerting based on missing heartbeats unreliable.
According to the Microsoft support this is all fine, because it is in the SLA of 6hours!
"We guarantee that at least 99.9% of the time, log data will be indexed within six hours of the data being queued for indexing by the Operations Management Suite Log Analytics Service."
You can check this here: https://azure.microsoft.com/en-in/support/legal/sla/log-analytics/v1_1/
Nov 05 2017 06:58 PM
Overall Log Analytics is meant for warm path. Currently we have preview of near real time metrics (sub minute) for alerting purposes. If you are interested please see https://azure.microsoft.com/en-us/blog/get-alerts-faster-with-near-real-time-alerting-for-azure-plat...
Nov 05 2017 11:30 PM
Apr 15 2018 12:10 PM
Also had a client that experienced latency in ingestion for among other things Heartbeat. Currently Azure Alert has a public preview where they will make the heartbeats sent from the agents available as a metric in Azure (<5 minutes latency). Log analytics will still ingest the heartbeats, however more slowly. Currently there are a limitation that you have to create a Azure Alert rule for each compute object, however that can be scripted with PowerShell. Look me up on Twitter (@ToreGroneng) and I will share the script with anyone interested. In the near future, the Alert team will address this, so that you can have a single rule for all your computer objects.