Final Update: Wednesday, 03 June 2020 05:26 UTC
We've confirmed that all systems are back to normal with no customer impact as of 06/02, 22:00 UTC. Our logs show the incident started on 06/01, 15:00 UTC and that during the 1 day and 7 hours that it took to resolve the issue, some customers may have experienced delayed alerts globally. Alerts will eventually fire, however they may be up to 20 minutes delayed.
-
Root Cause: Engineers determined that some instances of a back-end service became unhealthy after reaching an operational threshold, causing throttling on a dependent service. This throttling resulted in the alert delays.
- Incident Timeline: 1 Day &7 Hours - 06/01, 15:00 UTC through 06/02, 22:00 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.
-Anmol