Experiencing Alerting failure for Log Search Alerts - 02/21 - Resolved

[Azure-Monitor-Team]

Final Update: Sunday, 21 February 2021 21:04 UTC

We've confirmed that all systems are back to normal with no customer impact as of 02/21, 20:40 UTC. Our logs show the incident started on 02/21, 14:50 UTC and that during the 5 hours and 50 minutes that it took to resolve the issue 95% of customers in Southeast Australia region experienced latent or missed log alerts.

• Root Cause: The failure was due to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts.
• Incident Timeline: 5 Hours & 50 minutes - 02/21, 14:50 UTC through 02/21, 20:40 UTC

We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Jeff

---

Update: Sunday, 21 February 2021 19:46 UTC

Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and have removed them and scaled up the backend service to handle the backlog. The backlog is currently draining, alerts will continue to be impacted until we can complete this process.

• Work Around: None
• Next Update: Before 02/21 21:00 UTC

-Jeff

---

Update: Sunday, 21 February 2021 18:49 UTC

Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them as well as scale up the backend service to handle the backlog. Alerts will continue to be impacted until we can complete this process.

• Work Around: None
• Next Update: Before 02/21 20:00 UTC

-Jeff

---

Update: Sunday, 21 February 2021 17:30 UTC

Root cause has been isolated to a backend component becoming overloaded with an unexpectedly large number of alerts which was impacting the ability to process new alerts. To address this issue we have found the alerts generating a large quantity of noise and are working to reduce them. Alerts will continue to be impacted until we can complete this process.

• Work Around: None
• Next Update: Before 02/21 18:30 UTC

-Jeff

---

Initial Update: Sunday, 21 February 2021 16:32 UTC

We are aware of issues within Log Search Alerts in Australia Southeast region and are actively investigating. Some customers may experience delayed or missed Log Search Alerts.

• Work Around: None
• Next Update: Before 02/21 19:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Jeff

---