Final Update: Tuesday, 29 October 2019 01:26 UTC
We've confirmed that all systems are back to normal with no customer impact as of 10/29, 1:00 UTC. Our logs show the incident started on 10/27, 1:00 UTC and during the impact window, customers would have experienced varying latencies up to approximately 8 hours for their Activity Logs of category Policy and Administrative to become available in South Central US, West US 2, West US, East US 2, and Central US and corresponding alerts against these Activity Logs to fire.
-
Root Cause: The failure was due to a service reaching an operational threshold, which even when scaled out, took several hours to process the backlog of activity logs.
-
Incident Timeline: 48 Hours & 0 minutes - 10/27, 1:00 UTC through 10/29, 1:00 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.
-Matt
Update: Monday, 28 October 2019 23:50 UTC
The number of undelivered activity logs continues to decrease. However, customers may continue to see activity log alerts with several hours of latency. Our current ETA for when the queue will be completely cleared is approximately 4 hours, but engineers will continue working to increase the speed of processing.
- Work Around: None.
- Next Update: Before 10/29 02:00 UTC
-Matt
Update: Monday, 28 October 2019 21:31 UTC
Root cause has been isolated to throttling issues that were occurring in a dependent service which allowed a backlog of undelivered activity logs to form. Engineers have scaled out the service in paired regions to attempt to process the backlog and we are seeing a gradual reduction in latency. We do not currently have an estimate for when the queue will be fully cleared.
- Work Around: None.
- Next Update: Before 10/29 00:00 UTC
-Matt
Update: Monday, 28 October 2019 18:59 UTC
We continue to investigate issues within Activity Log Alerts. At this time, we understand customer impact to be limited to Activity Log Alerts configured to alert against Activity Logs originating in South Central US, West US 2, West US, East US 2, and Central US where the category of the Activity Log is Policy or Administrative. We currently have no estimate for resolution.
- Work Around: None.
- Next Update: Before 10/28 21:00 UTC
-Matt
Initial Update: Monday, 28 October 2019 18:17 UTC
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience latency for ARM-based Activity Log Alerts of up to 8 hours for all US regions.
- Work Around: None
- Next Update: Before 10/28 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Matt
