Final Update: Saturday, 09 November 2019 21:31 UTC
We've confirmed that all systems are back to normal as of 11/09, 21:31 UTC. Our logs show the incident started on 11/09, 09:36 UTC and that during the 11 hours 25 minutes that it took to resolve the issue customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules).
Root Cause: The failure was due to issue in one of our dependent services.
Incident Timeline: 11 Hours & 55 minutes - 11/09, 09:36 UTC through 11/09, 21:31 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.
-Sindhu
Update: Saturday, 09 November 2019 20:50 UTC
We continue to investigate issues within Log Search Alerts, Metric Alerts, Activity Log alerts and Smart detector alerts. Root cause is not fully understood at this time. Customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules). Initial findings indicate that the problem began at 11/09 ~09:36 UTC. We currently have no estimate for resolution.
Work Around: None
Next Update: Before 11/09 23:00 UTC
-Sindhu
Initial Update: Saturday, 09 November 2019 18:15 UTC
We are aware of issues within Log Search Alerts, Metric Alerts, Activity Log alerts and Smart detector alerts. We are actively investigating the issue. Customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules).
Work Around: None
Next Update: Before 11/09 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Sindhu
