Experiencing Data Latency in Azure portal for Activity Log Alerts - 06/04 - Resolved

[Azure-Monitor-Team]

Final Update: Thursday, 04 June 2020 19:20 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/4, 18:54 UTC. Our logs show the incident started on 6/4, 12:05 UTC and that during the 6 hours and 49 minutes that it took to resolve the issue customers experienced delayed log search alerts by up to 2.5 hours.

• Root Cause: The failure was due to the service reaching an operating limit as a result of large unexpected spike in alert notifications across regions.
• Incident Timeline: 6 Hours & 49 minutes - 6/4, 12:05 UTC through 6/4, 18:54 UTC
  

We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.

-Jeff

---

Update: Thursday, 04 June 2020 17:06 UTC

Root cause has been isolated to an extremely large spike in log search alerts across multiple regions which was impacting activity log alerts ability to process alerts in a timely manner. To address this issue we have scaled out the service and the primary incident causing the spike has been mitigated. Some customers may continue experience delayed log search alerts of up to 2 hours. We estimate 2-3 hours before timely log search alerts are being processed again.

• Work Around: None
• Next Update: Before 06/04 21:30 UTC

-Jeff

---

Initial Update: Thursday, 04 June 2020 13:18 UTC

We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. Alerts would have eventually fired.

• Work Around: None
• Next Update: Before 06/04 17:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol

---