Final Update: Tuesday, 29 September 2020 07:52 UTC
We've confirmed that all systems are back to normal with no customer impact as of 09/29, 07:00 UTC. Our logs show the incident started on 09/28, 23:00 UTC and that during the 8 hours that it took to resolve the issue some customers may have experienced delayed Activity Log alerts, however Alerts would still have fired during this time, but may have been delayed by up to 6 hours.
-
Root Cause: We determined that one of the back-end services using Log Alerts caused an unexpected large increase in Log Alert activity, which created a backlog and caused the delays
-
Incident Timeline: 8 Hours - 09/28, 23:00 UTC through 09/29, 07:00 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.
-Vamshi
Initial Update: Tuesday, 29 September 2020 06:47 UTC
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed Activity Log Alerts. Alerts would have eventually fired.
- Work Around: None
- Next Update: Before 09/29 09:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Vamshi
