Final Update: Thursday, 04 June 2020 19:20 UTC
We've confirmed that all systems are back to normal with no customer impact as of 6/4, 18:54 UTC. Our logs show the incident started on 6/4, 12:05 UTC and that during the 6 hours and 49 minutes that it took to resolve the issue customers experienced delayed log search alerts by up to 2.5 hours.
-Jeff
We've confirmed that all systems are back to normal with no customer impact as of 6/4, 18:54 UTC. Our logs show the incident started on 6/4, 12:05 UTC and that during the 6 hours and 49 minutes that it took to resolve the issue customers experienced delayed log search alerts by up to 2.5 hours.
- Root Cause: The failure was due to the service reaching an operating limit as a result of large unexpected spike in alert notifications across regions.
- Incident Timeline: 6 Hours & 49 minutes - 6/4, 12:05 UTC through 6/4, 18:54 UTC
-Jeff
Update: Thursday, 04 June 2020 17:06 UTC
Root cause has been isolated to an extremely large spike in log search alerts across multiple regions which was impacting activity log alerts ability to process alerts in a timely manner. To address this issue we have scaled out the service and the primary incident causing the spike has been mitigated. Some customers may continue experience delayed log search alerts of up to 2 hours. We estimate 2-3 hours before timely log search alerts are being processed again.
Root cause has been isolated to an extremely large spike in log search alerts across multiple regions which was impacting activity log alerts ability to process alerts in a timely manner. To address this issue we have scaled out the service and the primary incident causing the spike has been mitigated. Some customers may continue experience delayed log search alerts of up to 2 hours. We estimate 2-3 hours before timely log search alerts are being processed again.
- Work Around: None
- Next Update: Before 06/04 21:30 UTC
Initial Update: Thursday, 04 June 2020 13:18 UTC
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. Alerts would have eventually fired.
-Anmol
We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. Alerts would have eventually fired.
- Work Around: None
- Next Update: Before 06/04 17:30 UTC
-Anmol
Updated Jun 04, 2020
Version 3.0
