Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1095487%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20issue%20in%20Azure%20Portal%20for%20Many%20Data%20Types%20-%2001%2F07%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1095487%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2008%20January%202020%2005%3A13%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2001%2F08%2C%2003%3A50%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2001%2F07%2C%2020%3A04%20UTC%20and%20that%20during%20the%20~7%20hours%20and%2046%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20in%20South%20Central%20US%20may%20have%20received%20failure%20notifications%20when%20processing%20queries%20and%20difficulties%20accessing%20data%20in%20the%20Azure%20Portal.%20Customers%20may%20also%20have%20seen%20alerts%20not%20firing%20or%20delays%20in%20Log%20Search%20Alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20non-impacting%20maintenance%20window%20for%20one%20of%20our%20backend%20service%20exceeded%20the%20expected%20threshold.%3CBR%20%2F%3E%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%207%20Hours%20%26amp%3B%2046%20minutes%20-%2001%2F07%2C%2020%3A04%20UTC%20through%2001%2F08%2C%2003%3A50%20UTC%3CBR%20%2F%3E%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%2C%20Log%20Analytics%20and%20Log%20Search%20Alerts%20as%20critical%20services%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Mohini%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2007%20January%202020%2022%3A35%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20maintenance%20activity%20in%20the%20backend%20clusters%20which%20was%20impacting%20data%20access%2C%20query%20failures%20in%20Application%20Insights%20and%20Log%20Analytics%20for%20customers%20in%20South%20Central%20US.%20Customers%20will%20also%20be%20experiencing%20alerting%20delays%20or%20failures%20in%20Log%20Search%20Alerts.%26nbsp%3BWe%20shall%20provide%20the%20final%20update%20once%20we%20get%20confirmation%20from%20our%20backend%20team%20that%20the%20maintenance%20work%20is%20completed.%20We%20apologize%20for%20sending%20out%20resolved%20communications%20earlier%2C%20the%20scheduled%20maintenance%20had%20missed%20nodes.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3ANone%3CNONE%20or%3D%22%22%20details%3D%22%22%3E%3C%2FNONE%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F08%2003%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2007%20January%202020%2021%3A19%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2001%2F07%2C%2021%3A54%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2001%2F07%2C%2020%3A04%20UTC%20and%20that%20during%20the%201%20hour%20and%2050%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20in%20South%20Central%20US%20experienced%20issues%20with%20data%20access%2C%20query%20failures%20in%20Application%20Insights%20and%20alerting%20issues%20in%20Log%20Search%20Alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20one%20of%20the%20backend%20service.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%201%20hour%20and%2050%20minutes%20-%2001%2F07%2C%2020%3A04%20UTC%20through%2001%2F07%2C%2021%3A54%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Tuesday%2C%2007%20January%202020%2020%3A52%20UTCWe%20are%20aware%20of%20issues%20within%20Application%20Insights%20and%20are%20actively%20investigating.%20Some%20customers%20in%20South%20Central%20US%20may%20experience%20issues%20with%20Data%20Access%2C%20Query%20Failures%20and%20alerting%20failures%20in%20Log%20Search%20Alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%3CNONE%20or%3D%22%22%20details%3D%22%22%3E%3C%2FNONE%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F08%2001%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jayadev%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1095487%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EApplication%20Insights%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 08 January 2020 05:13 UTC

We've confirmed that all systems are back to normal with no customer impact as of 01/08, 03:50 UTC. Our logs show the incident started on 01/07, 20:04 UTC and that during the ~7 hours and 46 minutes that it took to resolve the issue some customers in South Central US may have received failure notifications when processing queries and difficulties accessing data in the Azure Portal. Customers may also have seen alerts not firing or delays in Log Search Alerts.
  • Root Cause: The failure was due to non-impacting maintenance window for one of our backend service exceeded the expected threshold.
  • Incident Timeline: 7 Hours & 46 minutes - 01/07, 20:04 UTC through 01/08, 03:50 UTC
We understand that customers rely on Application Insights, Log Analytics and Log Search Alerts as critical services and apologize for any impact this incident caused.

-Mohini

Update: Tuesday, 07 January 2020 22:35 UTC

Root cause has been isolated to maintenance activity in the backend clusters which was impacting data access, query failures in Application Insights and Log Analytics for customers in South Central US. Customers will also be experiencing alerting delays or failures in Log Search Alerts. We shall provide the final update once we get confirmation from our backend team that the maintenance work is completed. We apologize for sending out resolved communications earlier, the scheduled maintenance had missed nodes.
  • Work Around:None
  • Next Update: Before 01/08 03:00 UTC
-Jayadev

Update: Tuesday, 07 January 2020 21:19 UTC

We've confirmed that all systems are back to normal with no customer impact as of 01/07, 21:54 UTC. Our logs show the incident started on 01/07, 20:04 UTC and that during the 1 hour and 50 minutes that it took to resolve the issue some customers in South Central US experienced issues with data access, query failures in Application Insights and alerting issues in Log Search Alerts.
  • Root Cause: The failure was due to one of the backend service.
  • Incident Timeline: 1 hour and 50 minutes - 01/07, 20:04 UTC through 01/07, 21:54 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Jayadev

Initial Update: Tuesday, 07 January 2020 20:52 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers in South Central US may experience issues with Data Access, Query Failures and alerting failures in Log Search Alerts.
  • Work Around:
  • Next Update: Before 01/08 01:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jayadev