Final Update: Friday, 17 January 2020 04:25 UTC
We've confirmed that all systems are back to normal in the Asia East region with no customer impact as of 1/17, 3:33 UTC. Our logs show the incident started on 1/16, 3:15 UTC and that during the ~12 hours that it took to resolve the issue 30% of customers experienced delayed, misfired, or missing alerts.
-
Root Cause: The failure was due to an automated deployment failure to a service as a result of a bad service state.
- Incident Timeline: 12 Hours & 18 minutes - 1/16, 3:15 UTC through 1/17, 3:33 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.
-Jeff
Update: Thursday, 16 January 2020 22:42 UTC
We continue to investigate issues within Log Search Alerts in the East Asia region. Some customers continue to experience delayed, misfired, or missing alerts. The problem began at 1/16 15:15 UTC and has since been significantly reduced however we currently have no estimate for complete resolution.
- Next Update: Before 01/17 05:00 UTC
-Jeff
Update: Thursday, 16 January 2020 19:23 UTC
We continue to investigate issues within Log Search Alerts in the East Asia region. Root cause is not fully understood at this time. Some customers continue to experience delayed, misfired, or missing alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at <1/16 ~15:15 UTC>. We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 01/16 22:30 UTC
-Jeff
Initial Update: Thursday, 16 January 2020 17:18 UTC
We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience delayed alerts or miss some alerts.
- Work Around: None
- Next Update: Before 01/16 19:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jeff
