Home
%3CLINGO-SUB%20id%3D%22lingo-sub-890168%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20Issue%20in%20Azure%20portal%20for%20Log%20Analytics%20-%2010%2F02%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-890168%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2003%20October%202019%2003%3A55%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F3%2C%2003%3A45%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F2%2C%2023%3A45%20UTC%20and%20that%20during%20the%204%20hours%20that%20it%20took%20to%20resolve%20the%20issue%20any%20customers%20in%20USGov%20Virginia%20or%20services%20dependent%20on%20querying%20Azure%20Log%20Analytics%20data%20experienced%20an%20inability%20to%20query%20that%20data.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20middle-tier%20system%20authentication%20failure.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%204%20Hours%20%26amp%3B%200%20minutes%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20and%20associated%20services%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2003%20October%202019%2001%3A39%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20a%20middle%20tier%20authentication%20issue%20which%20is%20impacting%20the%20ability%20for%20customers%20and%20components%20to%20query%20log%20analytics%20data%20in%20USGov%20Virginia%20region.%26nbsp%3B%20Engineers%20are%20working%20on%20isolating%20and%20updating%20authentication%20information.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F03%2004%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2003%20October%202019%2000%3A23%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Analytics%20in%20the%20USGov%20Virginia%20region.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%26nbsp%3B%20Customers%20in%20the%20region%20will%20continue%20to%20experience%20errors%20including%20502%20when%20attempting%20to%20query%20Log%20Analytics%20data.%20Initial%20findings%20indicate%20that%20the%20problem%20began%20at%20%26lt%3B10%2F02%20~23%3A45%20UTC%26gt%3B.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F03%2002%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jeff%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2002%20October%202019%2023%3A56%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20failed%20queries%20for%20data.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%20at%20this%20time%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F03%2002%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jeff%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-890168%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 03 October 2019 03:55 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/3, 03:45 UTC. Our logs show the incident started on 10/2, 23:45 UTC and that during the 4 hours that it took to resolve the issue any customers in USGov Virginia or services dependent on querying Azure Log Analytics data experienced an inability to query that data.
  • Root Cause: The failure was due to a middle-tier system authentication failure.
  • Incident Timeline: 4 Hours & 0 minutes
We understand that customers rely on Azure Log Analytics and associated services as a critical service and apologize for any impact this incident caused.

-Jeff

Update: Thursday, 03 October 2019 01:39 UTC

Root cause has been isolated to a middle tier authentication issue which is impacting the ability for customers and components to query log analytics data in USGov Virginia region.  Engineers are working on isolating and updating authentication information.
  • Work Around: None
  • Next Update: Before 10/03 04:00 UTC
-Jeff

Update: Thursday, 03 October 2019 00:23 UTC

We continue to investigate issues within Log Analytics in the USGov Virginia region. Root cause is not fully understood at this time.  Customers in the region will continue to experience errors including 502 when attempting to query Log Analytics data. Initial findings indicate that the problem began at <10/02 ~23:45 UTC>. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 10/03 02:30 UTC
-Jeff

Initial Update: Wednesday, 02 October 2019 23:56 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers may experience failed queries for data.
  • Work Around: None at this time
  • Next Update: Before 10/03 02:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jeff