Final Update: Friday, 17 June 2022 05:54 UTC
We've confirmed that all systems are back to normal with no customer impact as of 06/17, 05:44 UTC. Our logs show the incident started on 06/16, 14:05 UTC and that during the 15 hours & 39 minutes that it took to resolve the issue . Some of the customers might have experienced issues with missed alerts or missed alerts notifications in West Europe region.
-
Root Cause: The failure was due to backend dependencies become unhealthy.
-
Incident Timeline: 15 Hours & 39 minutes - 06/16, 14:05 UTC through 06/17, 05:44 UTC.
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.
-Srikanth
Update: Friday, 17 June 2022 01:25 UTC
We've identified a code regression in a recent deployment has caused the missing alerts and latency issues to occur. We were unable to rollback this recent deployment so we've begun work on a hotfix in order to mitigate the issue which will take approximately 4 hours to complete before we begin its deployment. The next update will be provided in 4 hours or as events warrant.
- Work Around:
- Next Update: Before 06/17 05:30 UTC
-Shwetha
Update: Thursday, 16 June 2022 21:56 UTCWe've identified a code regression in a recent deployment has caused the missing alerts and latency issues to occur. We were unable to rollback this recent deployment so we've begun work on a hotfix in order to mitigate the issue which will take approximately 3 hours to complete before we begin its deployment. The next update will be provided in 3 hours or as events warrant.
- Next Update: Before 06/17 02:00 UTC
-Robin Paul
Update: Thursday, 16 June 2022 20:12 UTC
We've identified a code regression in a recent deployment that has caused the missing alerts and latency issues to occur. We're working on rolling back this recent deployment in order to mitigate the issue.- Next Update: Before 06/16 22:30 UTC
-Robin Paul
Update: Thursday, 16 June 2022 18:03 UTC
We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers continue to experience missed alerts or missed alerts notifications in West Europe region. We are working to establish the start time for the issue, initial findings indicate that the problem began at 6/16 ~14:05 UTC. We currently have no estimate for resolution.
- Next Update: Before 06/16 20:30 UTC
-Robin Paul
Update: Thursday, 16 June 2022 15:45 UTCWe are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience issues with missed alerts or missed alerts notifications in West Europe region.
- Work Around:
- Next Update: Before 06/16 20:00 UTC
-Srikanth
