Degraded experience viewing metric alerts - 05/31 - Resolved

Published May 31 2022 04:36 PM 1,036 Views
Final Update: Wednesday, 01 June 2022 05:45 UTC

We've confirmed that all systems are back to normal with no customer impact as of 06/01, 05:30 UTC. Our logs show the incident started on 05/19, 23:00 UTC and that during the 12 days 6 hours & 30 minutes that it took to resolve the issue. Some of the customers might have experienced viewing two alerts for notification. When a resource is unhealthy, two notifications are being raised and similarly for alert resolution as well.

An additional symptom in some instances is that one or both of the duplicate alerts would not have been resolved.

  • Root Cause: The failure was due to one of the backend dependency.
  • Incident Timeline:  12 days 6 Hours & 30 minutes - 05/19, 23:00 UTC through 06/01, 05:30 UTC.
We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.

-Srikanth

Update: Tuesday, 31 May 2022 23:28 UTC

Root cause has been isolated to a bad deployment. To address this issue we are in the process of rolling back. As a customer of Metric Alerts you could be experiencing viewing two alerts for unhealthy resources. When a resource is unhealthy, two notifications are being raised and similarly for alert resolution, two notifications are being sent
  • Work Around:
  • Start time: 5/19/2022
  • Next Update: Before 06/01 05:30 UTC
-Eric Singleton

Version history
Last update:
‎Jun 02 2022 05:05 PM
Updated by: