Experiencing Alerting failure for Log Search Alerts - 05/19 - Resolved

Published May 19 2021 09:42 AM 570 Views
Final Update: Wednesday, 19 May 2021 15:58 UTC

We've confirmed that all systems are back to normal with no customer impact as of 05/19, 15:45 UTC. Our logs show the incident started on 05/19, 14:30 UTC and that during the 1 hour 15 minutes that it took to resolve the issue customers may have experienced issues with missed or delayed alerts in Fairfax.
  • Root Cause: Engineering team has determined that part of the backend workflow for processing Log Search Alerts became unhealthy after it had reached an operational threshold.
  • Incident Timeline: 1 Hour & 15 minutes - 05/19, 14:30 UTC through 05/19, 15:45 UTC

We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Vincent


%3CLINGO-SUB%20id%3D%22lingo-sub-2368936%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20-%2005%2F19%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2368936%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2019%20May%202021%2015%3A58%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CSPAN%20style%3D%22font-family%3A%20%26quot%3BSegoe%20UI%26quot%3B%2C%20system-ui%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20sans-serif%3B%22%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CSPAN%20style%3D%22font-family%3A%20Helvetica%3B%22%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2005%2F19%2C%2015%3A45%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2005%2F19%2C%2014%3A30%20UTC%20and%20that%20during%20the%201%20hour%2015%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20may%20have%20experienced%20issues%20with%20missed%20or%20delayed%20alerts%20in%20Fairfax.%3C%2FSPAN%3E%3C%2FDIV%3E%3CUL%3E%3CLI%20style%3D%22font-size%3A%2014px%3B%22%3E%3CSPAN%20style%3D%22font-family%3A%20Helvetica%3B%22%3E%3CU%20style%3D%22%22%3ERoot%20Cause%3C%2FU%3E%3A%20Engineering%20team%20has%20determined%20that%20part%20of%20the%20backend%20workflow%20for%20processing%20Log%20Search%20Alerts%20became%20unhealthy%20after%20it%20had%20reached%20an%20operational%20threshold.%3C%2FSPAN%3E%3C%2FLI%3E%3CLI%20style%3D%22font-size%3A%2014px%3B%22%3E%3CSPAN%20style%3D%22font-family%3A%20Helvetica%3B%22%3E%3CU%20style%3D%22%22%3EIncident%20Timeline%3C%2FU%3E%3A%201%20Hour%20%26amp%3B%2015%20minutes%20-%2005%2F19%2C%2014%3A30%20UTC%20through%2005%2F19%2C%2015%3A45%20UTC%3C%2FSPAN%3E%3C%2FLI%3E%3C%2FUL%3E%3CP%20style%3D%22font-size%3A%2014px%3B%22%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Vincent%3CBR%20%2F%3E%3C%2FP%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎May 19 2021 09:42 AM
Updated by: