Experiencing Alerting failure for Log Search Alerts - 03/20 - Resolved

[Azure-Monitor-Team]

Final Update: Friday, 20 March 2020 20:27 UTC

We've confirmed that all systems are back to normal with no customer impact as of 03/20, 19:40 UTC. Our logs show the incident started on 03/20, 11:35 UTC and that during the 8 hours and 5 minutes that it took to resolve the issue customers in the US Gov Regions may have experienced delayed or missed alerts.

• Root Cause: The failure was due to an incorrect configuration on a dependent backend service.
• Incident Timeline: 8 Hours & 5 minutes - 03/20, 11:35 UTC through 03/20, 19:40 UTC

We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Eric Singleton

---

Update: Friday, 20 March 2020 18:35 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers may continue to experience delayed or missed alerts in US Gov Regions. 

• Next Update: Before 03/20 21:00 UTC

-Eric Singleton

---

Update: Friday, 20 March 2020 16:45 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers may continue to experience delayed or missed alerts in US Gov regions. 

• Next Update: Before 03/20 19:00 UTC

-Eric Singleton

---

Initial Update: Friday, 20 March 2020 13:47 UTC

We are aware of issues within Log Search Alerts service and are actively investigating. Some customers may experience delayed or missed alerts issue in US Gov regions.

• Work Around: None
• Next Update: Before 03/20 16:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol

---