Final Update: Friday, 29 April 2022 22:37 UTC
We've confirmed that all systems are back to normal with no customer impact as of 04/29, 21:30 UTC. Our logs show the incident started on 04/29, 14:00 UTC and that during the 7 hours and 30 minutes that it took to resolve the issue some of the customers experienced latency in getting notifications from Activity Log Alerts.
We've confirmed that all systems are back to normal with no customer impact as of 04/29, 21:30 UTC. Our logs show the incident started on 04/29, 14:00 UTC and that during the 7 hours and 30 minutes that it took to resolve the issue some of the customers experienced latency in getting notifications from Activity Log Alerts.
- Root Cause: A backend service had reached an operational threshold causing notification latency to occur.
- Incident Timeline: 7 Hours & 30 minutes - 04/29, 14:00 UTC through 04/29, 21:30 UTC
-Saika
Update: Friday, 29 April 2022 19:40 UTC
Root cause has been isolated to sudden spike in large number of events overloaded the machines causing the delay in Activity Log Alerts. To address this issue, we scaled out the number of instances of this service. Some customers may experience latency in getting notifications from Activity Log Alerts.
Root cause has been isolated to sudden spike in large number of events overloaded the machines causing the delay in Activity Log Alerts. To address this issue, we scaled out the number of instances of this service. Some customers may experience latency in getting notifications from Activity Log Alerts.
- Work Around: None
- Next Update: Before 04/29 22:00 UTC
Updated Apr 29, 2022
Version 2.0