%3CLINGO-SUB%20id%3D%22lingo-sub-1785474%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20in%20East%20US%20region%20-%2010%2F15%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1785474%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2015%20October%202020%2019%3A49%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F15%2C%2019%3A40%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F15%2C%2018%3A00%20UTC%20and%20that%20during%20the%20one%20hour%20and%20forty%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%201.1%25%20of%20customers%20in%20the%20East%20US%20region%20experienced%20alert%20failures%2C%20delayed%20alerts%20and%20query%20failures.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20back%20end%20component%20becoming%20unresponsive%20due%20to%20unexpectedly%20heavy%20load.%20The%20component%20self-healed.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%201%20Hour%20%26amp%3B%2040%20minutes%20-%2010%2F15%2C%2018%3A00%20UTC%20through%2010%2F15%2C%2019%3A40%20UTC%3CBR%20%2F%3E%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2015%20October%202020%2019%3A17%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20Log%20Analytics%20query%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20delayed%20or%20failed%20alerts%20as%20well%20as%20errors%20while%20querying%20Log%20Analytics%20data%20in%20the%20Azure%20portal.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CBR%20%2F%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F15%2020%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jack%20Cantwell%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1785474%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 15 October 2020 19:49 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/15, 19:40 UTC. Our logs show the incident started on 10/15, 18:00 UTC and that during the one hour and forty minutes that it took to resolve the issue 1.1% of customers in the East US region experienced alert failures, delayed alerts and query failures.
  • Root Cause: The failure was due to a back end component becoming unresponsive due to unexpectedly heavy load. The component self-healed.
  • Incident Timeline: 1 Hour & 40 minutes - 10/15, 18:00 UTC through 10/15, 19:40 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Jack Cantwell

Initial Update: Thursday, 15 October 2020 19:17 UTC

We are aware of issues within Log Search Alerts and Log Analytics query and are actively investigating. Some customers may experience delayed or failed alerts as well as errors while querying Log Analytics data in the Azure portal.

  • Next Update: Before 10/15 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jack Cantwell