Experiencing Missed Alerts for Log Search Alerts - 06/16 - Resolved

Published Jun 16 2022 08:50 AM 587 Views
Final Update: Friday, 17 June 2022 05:54 UTC

We've confirmed that all systems are back to normal with no customer impact as of 06/17, 05:44 UTC. Our logs show the incident started on 06/16, 14:05 UTC and that during the 15 hours & 39 minutes that it took to resolve the issue . Some of the customers might have experienced issues with missed alerts or missed alerts notifications in West Europe region.
  • Root Cause: The failure was due to backend dependencies become unhealthy.
  • Incident Timeline: 15 Hours & 39 minutes - 06/16, 14:05 UTC through 06/17, 05:44 UTC.
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Srikanth

Update: Friday, 17 June 2022 01:25 UTC

We've identified a code regression in a recent deployment has caused the missing alerts and latency issues to occur. We were unable to rollback this recent deployment so we've begun work on a hotfix in order to mitigate the issue which will take approximately 4 hours to complete before we begin its deployment. The next update will be provided in 4 hours or as events warrant.
  • Work Around:
  • Next Update: Before 06/17 05:30 UTC
-Shwetha

Update: Thursday, 16 June 2022 21:56 UTC

We've identified a code regression in a recent deployment has caused the missing alerts and latency issues to occur. We were unable to rollback this recent deployment so we've begun work on a hotfix in order to mitigate the issue which will take approximately 3 hours to complete before we begin its deployment. The next update will be provided in 3 hours or as events warrant.

  • Next Update: Before 06/17 02:00 UTC
-Robin Paul

Update: Thursday, 16 June 2022 20:12 UTC

We've identified a code regression in a recent deployment that has caused the missing alerts and latency issues to occur. We're working on rolling back this recent deployment in order to mitigate the issue.

  • Next Update: Before 06/16 22:30 UTC
-Robin Paul

Update: Thursday, 16 June 2022 18:03 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers continue to experience missed alerts or missed alerts notifications in West Europe region. We are working to establish the start time for the issue, initial findings indicate that the problem began at 6/16 ~14:05 UTC. We currently have no estimate for resolution.
  • Next Update: Before 06/16 20:30 UTC
-Robin Paul

Update: Thursday, 16 June 2022 15:45 UTC
We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience issues with missed alerts or missed alerts notifications in West Europe region.
  • Work Around:
  • Next Update: Before 06/16 20:00 UTC
-Srikanth

Version history
Last update:
‎Jun 16 2022 11:11 PM
Updated by: