Experiencing Data Gaps issue in Azure Portal for Many Data Types - 04/26 - Resolved

Published Apr 26 2021 01:59 PM 751 Views
Final Update: Tuesday, 27 April 2021 00:04 UTC

We've confirmed that all systems are back to normal with no customer impact as of 04/26, 23:20 UTC. Our logs show the incident started on 04/26, 20:20 UTC and that during the 3 hours that it took to resolve the issue some customers using Application Insights in West US 2 Region may have experienced intermittent metric data gaps and incorrect alert activation. Customers using Log Analytics Workspace-based Application Insights resources may experience log data gaps and incorrect alert activation. Additionally, customers using Custom Metrics (Preview) may experience data gaps and incorrect alert activation.
  • Root Cause: The failure was due to issues with one of the backend services.
  • Incident Timeline: 3 Hours - 04/26, 20:20 UTC through 04/26, 23:20 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Jayadev

Update: Monday, 26 April 2021 22:42 UTC

Root cause has been isolated to data loss for metric data in Application Insights. We found a root cause where the dependent service responsible for processing this data had a storage account which became unhealthy and unavailable to process any data. We are recreating this storage account, and once it is validated, this issue should be mitigated.

  • Work Around: none
  • Next Update: Before 04/27 01:00 UTC
-Ian

Initial Update: Monday, 26 April 2021 20:56 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers in West US 2 may experience data loss for metric data. This may cause failed or misfired alerts.
  • Work Around: none
  • Next Update: Before 04/26 23:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Ian

%3CLINGO-SUB%20id%3D%22lingo-sub-2294943%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Gaps%20issue%20in%20Azure%20Portal%20for%20Many%20Data%20Types%20-%2004%2F26%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2294943%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2027%20April%202021%2000%3A04%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2004%2F26%2C%2023%3A20%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2004%2F26%2C%2020%3A20%26nbsp%3BUTC%20and%20that%20during%20the%203%20hours%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20using%20Application%20Insights%20in%20West%20US%202%20Region%20may%20have%20experienced%20intermittent%20metric%20data%20gaps%20and%20incorrect%20alert%20activation.%20Customers%20using%20Log%20Analytics%20Workspace-based%20Application%20Insights%20resources%20may%20experience%20log%20data%20gaps%20and%20incorrect%20alert%20activation.%20Additionally%2C%20customers%20using%20Custom%20Metrics%20(Preview)%20may%20experience%20data%20gaps%20and%20incorrect%20alert%20activation.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20issues%20with%20one%20of%20the%20backend%20services.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%203%20Hours%20-%2004%2F26%2C%2020%3A20%26nbsp%3BUTC%20through%2004%2F26%2C%2023%3A20%26nbsp%3BUTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Monday%2C%2026%20April%202021%2022%3A42%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20data%20loss%20for%20metric%20data%20in%20Application%20Insights.%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3EWe%20found%20a%20root%20cause%20where%20the%20dependent%20service%20responsible%20for%20processing%20this%20data%20had%20a%20storage%20account%20which%20became%20unhealthy%20and%20unavailable%20to%20process%20any%20data.%20We%20are%20recreating%20this%20storage%20account%2C%20and%20once%20it%20is%20validated%2C%20this%20issue%20should%20be%20mitigated.%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F27%2001%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Ian%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Monday%2C%2026%20April%202021%2020%3A56%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Application%20Insights%20and%20are%20actively%20investigating.%20Some%20customers%20in%20West%20US%202%20may%20experience%20data%20loss%20for%20metric%20data.%20This%20may%20cause%20failed%20or%20misfired%20alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F26%2023%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Ian%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Apr 26 2021 05:28 PM
Updated by: