We've confirmed that all systems are back to normal with no customer impact as of 04/29, 21:30 UTC. Our logs show the incident started on 04/29, 14:00 UTC and that during the 7 hours and 30 minutes that it took to resolve the issue some of the customers experienced latency in getting notifications from Activity Log Alerts.
Root Cause: A backend service had reached an operational threshold causing notification latency to occur.
Incident Timeline: 7 Hours & 30 minutes - 04/29, 14:00 UTC through 04/29, 21:30 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.
Update: Friday, 29 April 2022 19:40 UTC
Root cause has been isolated to sudden spike in large number of events overloaded the machines causing the delay in Activity Log Alerts. To address this issue, we scaled out the number of instances of this service. Some customers may experience latency in getting notifications from Activity Log Alerts.