Experiencing Alerting failure for Metric Alerts - 10/02 - Resolved

[Azure-Monitor-Team]

Final Update: Wednesday, 02 October 2019 17:11 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/2, 16:40 UTC. Our logs show the incident started on 10/2, 01:45 UTC and that during that time customers would have experienced intermittent missing of configured metric alerts.

• Root Cause: The failure was due to performance bug in a backend system. 
• Incident Timeline: 14 Hours & 55 minutes

We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.

-Jeff

---

Update: Wednesday, 02 October 2019 15:23 UTC

Root cause has been isolated to failure in one of our dependent services which was impacting Metric alerts. We continue to work on fixing the issue. Some customers in East US region may still continue to experience issues with Metric Alerts either not being delivered or may receive false positive alerts. 

• Next Update: Before 10/02 19:30 UTC

-Madhuri

---

Update: Wednesday, 02 October 2019 09:22 UTC

We continue to investigate issues within Metric Alerts. Root cause is not fully understood at this time. Some customers in East US region may still continue to experience issues with Metric Alerts either not being delivered or may receive false positive alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 10/02 01:45 UTC. 

• Next Update: Before 10/02 15:30 UTC

-Madhuri

---

Initial Update: Wednesday, 02 October 2019 05:23 UTC

We are aware of issues within Metric Alerts and are actively investigating. Some customers in East US region may experience issues with Metric Alerts either not being delivered or may receive false alerts.

• Next Update: Before 10/02 09:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Madhuri

---