Final Update: Wednesday, 03 June 2020 05:26 UTC
We've confirmed that all systems are back to normal with no customer impact as of 06/02, 22:00 UTC. Our logs show the incident started on 06/01, 15:00 UTC and that during the 1 day and 7 hours that it took to resolve the issue, some customers may have experienced delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed.
-Anmol
We've confirmed that all systems are back to normal with no customer impact as of 06/02, 22:00 UTC. Our logs show the incident started on 06/01, 15:00 UTC and that during the 1 day and 7 hours that it took to resolve the issue, some customers may have experienced delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed.
- Root Cause: Engineers determined that some instances of a back-end service became unhealthy after reaching an operational threshold, causing throttling on a dependent service. This throttling resulted in the alert delays.
- Incident Timeline: 1 Day &7 Hours - 06/01, 15:00 UTC through 06/02, 22:00 UTC
-Anmol
Update: Tuesday, 02 June 2020 17:06 UTC
We continue to investigate issues within Activity Log Alerts. Root cause has been narrowed to a dependent service throttling issue which we are working on resolving. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed. The problem began at 6/01 15:00 UTC. We currently have no estimate for resolution.
We continue to investigate issues within Activity Log Alerts. Root cause has been narrowed to a dependent service throttling issue which we are working on resolving. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed. The problem began at 6/01 15:00 UTC. We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 06/03 05:00 UTC
Update: Tuesday, 02 June 2020 12:59 UTC
We continue to investigate issues within Activity Log Alerts. Root cause is not fully understood at this time. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed. We are working to establish the start time for the issue, initial findings indicate that the problem began at 05/01 ~15:00 UTC. We currently have no estimate for resolution.
We continue to investigate issues within Activity Log Alerts. Root cause is not fully understood at this time. Some customers continue to experience delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed. We are working to establish the start time for the issue, initial findings indicate that the problem began at 05/01 ~15:00 UTC. We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 06/02 17:00 UTC
Initial Update: Tuesday, 02 June 2020 10:20 UTC
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts.
-Anmol
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts.
- Work Around: None
- Next Update: Before 06/02 13:30 UTC
-Anmol
Updated Jun 02, 2020
Version 4.0