Final Update: Friday, 09 August 2019 23:07 UTC
We've confirmed that all systems are back to normal with no customer impact as of 8/9, 22:54 UTC. Our logs show the incident started on 8/9, 18:59 UTC and that during the 3 hours that it took to resolve the issue 1062 subscriptions (~16%) experienced delayed alerts that fired after the issue was mitigated.
-Ian
We've confirmed that all systems are back to normal with no customer impact as of 8/9, 22:54 UTC. Our logs show the incident started on 8/9, 18:59 UTC and that during the 3 hours that it took to resolve the issue 1062 subscriptions (~16%) experienced delayed alerts that fired after the issue was mitigated.
- Root Cause: The failure was due to a burst of service health activity log events which increased the evaluation latency of the events. Scaling out the service mitigated the issue.
- Incident Timeline: 3 Hours & 55 minutes - 8/9, 18:59 UTC through 8/9, 22:54 UTC
-Ian
Updated Aug 09, 2019
Version 2.0