I'm trying to use OMS alerts for heartbeat of my servers. On the old Log analytics, i readed an article saying thats the agent reports each 60 seconds for the OMS and the results of the queries is normally, obeying the frequency of agent reports.
After migration to the new Log Analytics, my query return a great delay on the registers of the heartbeats, having 30 minutes of late for the registers answered.
Anyone could clarify if we have some changes or trouble on the agents or the results of queries on the new Log Analytics?
I've noticed a little bit of a larger delay in event and heartbeat data being recorded in OMS - in some testing I saw heartbeat data taking up to 8 minutes to show up in log searches, where it had previously usually taken at most 3 minutes. The heartbeat data is still recorded every 60 seconds - but the indexing and displaying in the log search can take longer.
You will get inconsistant alerts. With the new query langauge you have to specify the sorting. In the old one sorting was always done on TimeGenerated if not specified. So you have to do query like this to get the latest data
Heartbeat | sort by TimeGenerated desc
Otherwise there is no change on how logs are ingested into Log Analytics.
I've noticed the same problem in my workspace. It doesn’t matter if you sort your query results by TimeGenerated or not. The Heartbeats are just late in some cases... Sometimes up to 20 minutes, which makes alerting based on missing heartbeats unreliable.
According to the Microsoft support this is all fine, because it is in the SLA of 6hours!
"We guarantee that at least 99.9% of the time, log data will be indexed within six hours of the data being queued for indexing by the Operations Management Suite Log Analytics Service."
Also had a client that experienced latency in ingestion for among other things Heartbeat. Currently Azure Alert has a public preview where they will make the heartbeats sent from the agents available as a metric in Azure (<5 minutes latency). Log analytics will still ingest the heartbeats, however more slowly. Currently there are a limitation that you have to create a Azure Alert rule for each compute object, however that can be scripted with PowerShell. Look me up on Twitter (@ToreGroneng) and I will share the script with anyone interested. In the near future, the Alert team will address this, so that you can have a single rule for all your computer objects.