Final Update: Wednesday, 05 June 2019 23:58 UTC
We've confirmed that all systems are back to normal with no customer impact as of June 5th 2019, 21:36 UTC. Our logs show the incident started on June 5th 2019, 15:00 UTC and that during the time that it took to resolve the issue subset of customers would have missed receiving Alerting notifications. Customers might be receiving these missed notifications in a delayed timeline as the system processes the backlog.
-chandar.
We've confirmed that all systems are back to normal with no customer impact as of June 5th 2019, 21:36 UTC. Our logs show the incident started on June 5th 2019, 15:00 UTC and that during the time that it took to resolve the issue subset of customers would have missed receiving Alerting notifications. Customers might be receiving these missed notifications in a delayed timeline as the system processes the backlog.
- Root Cause: The failure was due to a performance issues of a downstream system
- Lessons Learned: Chance of recurrence of this issue is low, and we will continue to monitor the system.
- Incident Timeline: June 5th 2019, 21:36 UTC through June 5th 2019, 15:00 UTC
-chandar.
Update: Wednesday, 05 June 2019 18:44 UTC
Root cause has been isolated to downstream service which was impacting alerting notification for subset of customers. To address this issue we are in the process of rolling out a fix. Some customers may experience notification failures and we estimate 1 hour before all notification impacts are addressed.
Root cause has been isolated to downstream service which was impacting alerting notification for subset of customers. To address this issue we are in the process of rolling out a fix. Some customers may experience notification failures and we estimate 1 hour before all notification impacts are addressed.
- Work Around: none
- Next Update: Before 06/05 21:00 UTC
Updated Jun 06, 2019
Version 2.0