Final Update: Wednesday, 18 December 2019 15:41 UTC
We've confirmed that all systems are back to normal with no customer impact as of 12/18, 15:15 UTC. Our logs show the incident started on 12/13, 13:50 UTC and that during the 5 days 1 hours & 25 minutes that it took to resolve the issue ~45 customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events.
-
Root Cause: The failure was due to an issue with one of our dependent service.
- Incident Timeline: 5 Days 1 Hours & 25 minutes - 12/13, 13:50 UTC through 12/18, 15:15 UTC
We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.
-Anmol
Update: Wednesday, 18 December 2019 13:30 UTC
We continue to investigate issues within Metric Alerts. Some customers in US Sovereign Cloud using Classic Alert Rules and Autoscale may continue to experience missing email notifications for alerts that were fired or autoscale events. We are working to establish the start time for the issue, initial findings indicate that the problem began at 12/13 10:00 UTC. We currently have no estimate for resolution.
- Work Around: None
- Next Update: Before 12/18 17:30 UTC
-Anmol
Initial Update: Wednesday, 18 December 2019 10:45 UTC
We are aware of issues within Metric Alerts and are actively investigating. Some customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events.
- Work Around: None
- Next Update: Before 12/18 13:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol
