Experiencing Data Latency in Azure portal for Activity Log Alerts - 06/02 - Resolved

[Azure-Monitor-Team]

Final Update: Wednesday, 03 June 2020 05:26 UTC

We've confirmed that all systems are back to normal with no customer impact as of 06/02, 22:00 UTC. Our logs show the incident started on 06/01, 15:00 UTC and that during the 1 day and 7 hours that it took to resolve the issue, some customers may have experienced delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed.

• Root Cause: Engineers determined that some instances of a back-end service became unhealthy after reaching an operational threshold, causing throttling on a dependent service. This throttling resulted in the alert delays.
• Incident Timeline: 1 Day  &7 Hours  - 06/01, 15:00 UTC through 06/02, 22:00 UTC
  

We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.

-Anmol

---

Update: Tuesday, 02 June 2020 17:06 UTC

We continue to investigate issues within Activity Log Alerts. Root cause has been narrowed to a dependent service throttling issue which we are working on resolving. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed. The problem began at 6/01 15:00 UTC. We currently have no estimate for resolution.

• Work Around: None
• Next Update: Before 06/03 05:00 UTC

-Jeff

---

Update: Tuesday, 02 June 2020 12:59 UTC

We continue to investigate issues within Activity Log Alerts. Root cause is not fully understood at this time. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed.  We are working to establish the start time for the issue, initial findings indicate that the problem began at 05/01 ~15:00 UTC. We currently have no estimate for resolution.

• Work Around: None
• Next Update: Before 06/02 17:00 UTC

-Anmol

---

Initial Update: Tuesday, 02 June 2020 10:20 UTC

 We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. 

• Work Around: None
• Next Update: Before 06/02 13:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol

---