Final Update: Friday, 11 October 2019 09:08 UTC
We've confirmed that all systems are back to normal with no customer impact as of 10/03, 16:45 UTC. Our logs show the incident started on 08/24, 14:06 UTC and that during the ~936 hours that it took to resolve the issue some of customers experienced issues with classic alert rules not delivering notifications.
- Root Cause: The failure was due to issue in one of our backend service.
- Incident Timeline: 936 Hours - 08/24, 14:06 UTC through 10/03, 16:45 UTC
We understand that customers rely on Metric alerts as a critical service and apologize for any impact this incident caused.
-Naresh
Update: Wednesday, 25 September 2019 22:45 UTC
Root cause has been identified as a code-related issue impacting classic alert rules that were configured prior to 2017. To address the issue a hotfix has been tested and validated in the pre-production environment. As the fix is somewhat complex, it will be deployed to all regions over the next 5 business days. Some customers may continue to experience issues with classic alert rules not delivering notifications until the deployment is completed to all regions.
-Jayadev
Update: Wednesday, 25 September 2019 21:11 UTC
We continue to investigate issues within Metric alerts.Root cause was determined to be an issue with alerts created prior to 2017 not having a required field.Some customers continue to experience issues with classic alert rules not delivering notifications.We are working to establish the start time for the issue,initial findings indicate that the problem began at 07/08/19 ~00:00 UTC.We currently have no estimate for resolution.
- Next Update: Before 09/26 09:30 UTC
-Jayadev
Update: Wednesday, 25 September 2019 17:16 UTC
We continue to investigate issues within Metric alerts.Root cause was determined to be an issue with alerts created prior to 2017 not having a required field.Some customers continue to experience issues with classic alert rules not delivering notifications.We are working to establish the start time for the issue,initial findings indicate that the problem began at 07/08/19 ~00:00 UTC.We currently have no estimate for resolution.- Next Update: Before 09/25 21:30 UTC
-Jayadev
Update: Wednesday, 25 September 2019 13:32 UTC
We continue to investigate issues within Metric alerts.Root cause was determined to be an issue with alerts created prior to 2017 not having a required field.Some customers continue to experience issues with classic alert rules not delivering notifications.We are working to establish the start time for the issue,initial findings indicate that the problem began at 07/08/19 ~00:00 UTC.We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 09/25 18:00 UTC
-Naresh
Update: Wednesday, 25 September 2019 06:13 UTC
We continue to investigate issues within Metric alerts.Root cause was determined to be an issue with alerts created prior to 2017 not having a required field.Some customers continue to experience issues with classic alert rules not delivering notifications.We are working to establish the start time for the issue,initial findings indicate that the problem began at 07/08/19 ~00:00 UTC.We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 09/25 12:30 UTC
-Naresh
Update: Tuesday, 24 September 2019 18:38 UTC
We continue to investigate issues within Metric alerts.Root cause was determined to be an issue with alerts created prior to 2017 not having a required field.Some customers continue to experience issues with classic alert rules not delivering notifications.We are working to establish the start time for the issue,initial findings indicate that the problem began at 07/08/19 ~00:00 UTC.We currently have no estimate for resolution.
- Next Update: Before 09/25 01:00 UTC
-Jayadev
Initial Update: Tuesday, 24 September 2019 15:03 UTC
We are aware of issues within Metric alerts and are actively investigating.Some customers may experience issue running classic alert rules.
- Work Around: None
- Next Update: Before 09/24 18:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Naresh
