Experienced Alerting failure for Log Search Alerts in China Gov- 01/01 - Resolved

Published Jan 01 2021 02:15 AM 1,260 Views
Final Update: Friday, 01 January 2021 15:42 UTC

We've confirmed that all systems are back to normal with no customer impact as of 01/01, 15:30 UTC. Our logs show the incident started on 01/01, 08:20 UTC and that during the 7 Hours 10 Minutes that it took to resolve the issue some of customers may have experienced issues with delayed or wrongly fired Log Search alerts or experienced difficulties accessing data for resources hosted in China East Region.
  • Root Cause: We identified the issue was caused by a configuration setting.
  • Incident Timeline: 7 Hours & 10 minutes - 01/01, 08:20 UTC through 01/01, 15:30 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Madhav

Update: Friday, 01 January 2021 13:27 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience issues with missed, delayed or wrongly fired Log Search Alerts or experience difficulties accessing data for resources hosted in China East 2 region. 
  • Root CauseThe failure was due to a recent deployment in one of the backend services. 
  • Next Update: Before 01/01 15:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Madhav

Initial Update: Friday, 01 January 2021 10:08 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience issues with missed, delayed or wrongly fired Log Search Alerts or experience difficulties accessing data for resources hosted in China East 2 region. 
  • Work Around: None
  • Next Update: Before 01/01 13:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol

%3CLINGO-SUB%20id%3D%22lingo-sub-2025261%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20in%20China%20Gov-%2001%2F01%20-%20Investigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2025261%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2001%20January%202021%2013%3A27%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20issues%20with%20missed%2C%20delayed%20or%20wrongly%20fired%20Log%20Search%20Alerts%20or%20experience%20difficulties%20accessing%20data%20for%20resources%20hosted%20in%20China%20East%202%20region.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%26nbsp%3B%3CSPAN%20style%3D%22color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%26quot%3BHelvetica%20Neue%26quot%3B%2C%20Helvetica%2C%20Arial%2C%20sans-serif%3B%22%3EThe%20failure%20was%20due%20to%20a%20recent%20deployment%20in%20one%20of%20the%20backend%20services.%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F01%2015%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E%3CP%3E%3CSPAN%20style%3D%22color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%26quot%3BHelvetica%20Neue%26quot%3B%2C%20Helvetica%2C%20Arial%2C%20sans-serif%3B%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3C%2FSPAN%3E%3CBR%20%2F%3E%3C%2FP%3E-Madhav%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Friday%2C%2001%20January%202021%2010%3A08%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20issues%20with%20missed%2C%20delayed%20or%20wrongly%20fired%20Log%20Search%20Alerts%20or%20experience%20difficulties%20accessing%20data%20for%20resources%20hosted%20in%20China%20East%202%20region.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F01%2013%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Anmol%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2025261%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Version history
Last update:
‎Jan 01 2021 07:54 AM
Updated by: