Home
%3CLINGO-SUB%20id%3D%22lingo-sub-715878%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20Issue%20in%20Azure%20portal%20for%20Log%20Analytics%20-%2006%2F24%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-715878%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Monday%2C%2024%20June%202019%2005%3A04%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%206%2F24%2C%2004%3A54%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%206%2F23%2C%2022%3A50%20UTC%20and%20that%20during%20the%206%20hours%20and%204%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%202%2C588%20customers%20experienced%20an%20inability%20to%20query%20their%20Log%20Analytics%20data%20and%20possible%20alert%20malfunctions.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20configuration%20change%20on%20the%20backend.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%206%20Hours%20%26amp%3B%204%20minutes%20-%206%2F23%2C%2022%3A50%20UTC%20through%206%2F24%2C%2004%3A54%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Monday%2C%2024%20June%202019%2004%3A49%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20an%20incorrect%20configuration%20on%20the%20backend%20which%20was%20impacting%20query%20times.%20To%20address%20this%20issue%20we%20have%20applied%20an%20update%20to%20the%20configuration.%26nbsp%3B%20We%20see%20queries%20now%20succeeding.%26nbsp%3B%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F24%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Monday%2C%2024%20June%202019%2003%3A59%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Analytics.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20errors%20when%20searching%20and%20possible%20misbehaving%20alerts.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2006%2F23%2022%3A50%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F24%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Monday%2C%2024%20June%202019%2002%3A15%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20in%20East%20Japan%20may%20experience%20issues%20querying%20data%20and%20alerts%20may%20be%20misfiring.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F24%2004%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jeff%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-715878%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Monday, 24 June 2019 05:04 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/24, 04:54 UTC. Our logs show the incident started on 6/23, 22:50 UTC and that during the 6 hours and 4 minutes that it took to resolve the issue 2,588 customers experienced an inability to query their Log Analytics data and possible alert malfunctions.
  • Root Cause: The failure was due to a configuration change on the backend.
  • Incident Timeline: 6 Hours & 4 minutes - 6/23, 22:50 UTC through 6/24, 04:54 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Jeff

Update: Monday, 24 June 2019 04:49 UTC

Root cause has been isolated to an incorrect configuration on the backend which was impacting query times. To address this issue we have applied an update to the configuration.  We see queries now succeeding. 
  • Next Update: Before 06/24 06:00 UTC
-Jeff

Update: Monday, 24 June 2019 03:59 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to experience errors when searching and possible misbehaving alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/23 22:50 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 06/24 06:00 UTC
-Jeff

Initial Update: Monday, 24 June 2019 02:15 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers in East Japan may experience issues querying data and alerts may be misfiring.
  • Work Around: none
  • Next Update: Before 06/24 04:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jeff