Experiencing Alerting failure for Log Search Alerts - 11/30 - Resolved

Published Nov 30 2021 08:27 AM 1,409 Views
Final Update: Tuesday, 30 November 2021 17:07 UTC

We've confirmed that all systems are back to normal with no customer impact as of 11/30, 17:00 UTC. Our logs show the incident started on 11:30, 14:30 UTC and that during the 2 hours and 30 minutes that it took to resolve the issue customers experienced either failed or misfired alerts.
  • Root Cause: The failure was due to a bad deployment causing alert rules to potentially not run.
  • Incident Timeline: 2 Hours & 30 minutes -11/30, 14:30 UTC through 11/30, 17:00 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Ian

Update: Tuesday, 30 November 2021 16:23 UTC

Alert rules for Log Search Alerts are failing in West Europe and East US. To address this issue we are rolling back to our latest update. Some customers may experience failed or misfired alerts.
  • Work Around: none
  • Next Update: Before 11/30 18:30 UTC
-Ian

%3CLINGO-SUB%20id%3D%22lingo-sub-3019029%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20-%2011%2F30%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3019029%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2030%20November%202021%2017%3A07%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2011%2F30%2C%2017%3A00%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2011%3A30%2C%2014%3A30%20UTC%20and%20that%20during%20the%202%20hours%20and%2030%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20experienced%20either%20failed%20or%20misfired%20alerts.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20bad%20deployment%20causing%20alert%20rules%20to%20potentially%20not%20run.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%202%20Hours%20%26amp%3B%2030%20minutes%20-11%2F30%2C%2014%3A30%20UTC%20through%2011%2F30%2C%2017%3A00%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Ian%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2030%20November%202021%2016%3A23%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EAlert%20rules%20for%20Log%20Search%20Alerts%20are%20failing%20in%20West%20Europe%20and%20East%20US.%20To%20address%20this%20issue%20we%20are%20rolling%20back%20to%20our%20latest%20update.%20Some%20customers%20may%20experience%20failed%20or%20misfired%20alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2011%2F30%2018%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Ian%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Nov 30 2021 09:10 AM
Updated by: