%3CLINGO-SUB%20id%3D%22lingo-sub-1733596%22%20slang%3D%22en-US%22%3EExperiencing%20Issues%20within%20Log%20Search%20Alert%20Rules%20-%2010%2F01%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1733596%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2001%20October%202020%2010%3A41%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F01%2C%2009%3A45%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F01%2C%2005%3A30%20UTC%20and%20that%20during%20the%204%20hours%2015%20Minutes%20that%20it%20took%20to%20resolve%20the%20issue.%20Some%20customers%20may%20have%20experienced%20failure%20notifications%20when%20accessing%20or%20performing%20service%20management%20operations%20such%20as%20Create%2C%20Update%2C%20Delete%20for%20Azure%20Log%20Search%20Alert%20Rules%20in%20multiple%20regions.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20performance%20degradation%20following%20a%20recent%20backend%20service%20deployment.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%204%20Hours%20%26amp%3B%2015%20minutes%20-%2010%2F01%2C%2005%3A30%20UTC%20through%2010%2F01%2C%2009%3A45%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Vamshi%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2001%20October%202020%2009%3A25%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20may%20experience%20failure%20notifications%20when%20accessing%20or%20performing%20service%20management%20operations%20such%20as%20Create%2C%20Update%2C%20Delete%20for%20Azure%20Log%20Search%20Alert%20Rules%20in%20multiple%20regions.%20Initial%20findings%20indicate%20that%20the%20problem%20began%20at%2010%2F01%20~05%3A30%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F01%2012%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Vamshi%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22%22%3E%3CU%20style%3D%22font-size%3A%2014px%3B%22%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2001%20October%202020%2008%3A11%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20failure%20notifications%20when%20accessing%20or%20performing%20service%20management%20operations%20such%20as%20Create%2C%20Update%2C%20Delete%20for%20Azure%20Log%20Search%20Alert%20Rules%20in%20East%20US%20Region.%3CBR%20%2F%3E%3CUL%20style%3D%22font-size%3A%2014px%3B%22%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%26nbsp%3B%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F01%2011%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Vamshi%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1733596%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 01 October 2020 10:41 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/01, 09:45 UTC. Our logs show the incident started on 10/01, 05:30 UTC and that during the 4 hours 15 Minutes that it took to resolve the issue. Some customers may have experienced failure notifications when accessing or performing service management operations such as Create, Update, Delete for Azure Log Search Alert Rules in multiple regions.
  • Root Cause: The failure was due to a performance degradation following a recent backend service deployment.
  • Incident Timeline: 4 Hours & 15 minutes - 10/01, 05:30 UTC through 10/01, 09:45 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Vamshi

Update: Thursday, 01 October 2020 09:25 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers may experience failure notifications when accessing or performing service management operations such as Create, Update, Delete for Azure Log Search Alert Rules in multiple regions. Initial findings indicate that the problem began at 10/01 ~05:30 UTC. We currently have no estimate for resolution.
  • Next Update: Before 10/01 12:30 UTC
-Vamshi

Initial Update: Thursday, 01 October 2020 08:11 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience failure notifications when accessing or performing service management operations such as Create, Update, Delete for Azure Log Search Alert Rules in East US Region.
  • Work Around:  None
  • Next Update: Before 10/01 11:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Vamshi