Experiencing Metric Alerts Issue for Log Analytics in East US region - 09/19 - Resolved

[Azure-Monitor-Team]

Final Update: Friday, 20 September 2019 10:33 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/20, 10:20 UTC. Our logs show the incident started on 09/19, 09:00 UTC and that during the 25 hours 20 minutes that it took to resolve the issue some customers in East US region might have experienced alerting delays and would not had received metric alerts.

• Root Cause: The failure was due to an issue in one of our dependent service.
• Incident Timeline: 25 Hours & 20 minutes - 09/19, 09:00 UTC through 09/20, 10:20 UTC

We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Monish

---

Update: Friday, 20 September 2019 07:41 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers in East Us Region continue to experience delayed or might not receive metric alerts.We currently have no estimate for resolution.

• Work Around:
• Next Update: Before 09/20 12:00 UTC

-Monish

---

Update: Friday, 20 September 2019 02:27 UTC

We continue to investigate issues within Log Analytics in East US region. Root cause is not fully understood at this time and we are working on resolution. Some customers continue to experience delay or may not be receiving metric alerts in East US. We currently have no estimate for resolution.

• Next Update: Before 09/20 07:30 UTC

-Jayadev

---

Update: Thursday, 19 September 2019 22:12 UTC

We continue to investigate issues within Log Analytics in East US region, we encountered another issue along the process of mitigation and we are working on resolution. Some customers continue to experience delay or may not be receiving alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 09/19 ~09:00 UTC. We currently have no estimate for resolution.

• Next Update: Before 09/20 02:30 UTC

-Jayadev

---

Update: Thursday, 19 September 2019 17:57 UTC

Root cause has been isolated to failure in deployment which was impacting metric alerts to be triggered for Log Analytics in East US region. To address this issue we scaled out our service for faster processing of data that was stuck. Data flow is now working as expected. Some customers may experience either delay or may not be receiving alerts and we estimate ~ 3 hours before all metric alerts start working as expected.

• Next Update: Before 09/19 22:00 UTC

-Jayadev

---