Experiencing Data Access issue in Azure Portal for Some Metrics - 08/31 - Resolved

Published Aug 30 2021 07:33 PM 683 Views
Final Update: Wednesday, 01 September 2021 16:40 UTC

We've confirmed that all systems are back to normal with no customer impact as of 2021-09-01 18:58 UTC. Our logs show the incident started on 2021-08-29 13:22 UTC and that during the two days, 5.5 hours that it took to resolve the issue 100% of customers who queried metrics from the Microsoft.Network/privateEndpoints and Microsoft.Network/privateLinkService resource providers will have returned 400 error codes and not been able to query their metrics.
  • Root Cause: The failure was due to new feature rollout that contained an error.
  • Lessons Learned: We will be applying greater monitoring in lower level environments to catch errors like these before they reach production environments.
  • Incident Timeline: 2 Days, 5 Hours & 36 minutes - 2021-08-29 13:22 UTC through 2021-09-01 18:58 UTC
We understand that customers rely on Azure Monitor and Azure Metrics as critical services and apologize for any impact this incident caused.

-Jack Cantwell

Update: Tuesday, 31 August 2021 02:26 UTC

Root cause has been isolated to a feature update which inadvertently caused metrics for the resource providers Microsoft.Network/privateEndpoints and Microsoft.Network/privateLinkService to return 400 errors when queried. To address this issue we are rolling out a hotfix. The hotfix rollout process across all regions will take approximately 18 hours. Some customers will continue to experience these issues until all regions have have gotten the hotfix applied.
  • Next Update: Before 08/31 20:30 UTC
-Jack Cantwell

%3CLINGO-SUB%20id%3D%22lingo-sub-2700901%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20issue%20in%20Azure%20Portal%20for%20Some%20Metrics%20-%2008%2F31%20-%20Mitigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2700901%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2031%20August%202021%2002%3A26%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20a%20feature%20update%20which%20inadvertently%20caused%20metrics%20for%20the%20resource%20providers%26nbsp%3B%3CSPAN%20style%3D%22color%3A%20rgb(67%2C%20104%2C%2042)%3B%20font-family%3A%20%26quot%3BMaiandra%20GD%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2011pt%3B%22%3EMicrosoft.Network%2FprivateEndpoints%20and%20Microsoft.Network%2FprivateLinkService%20to%20return%20400%20errors%20when%20queried%3C%2FSPAN%3E.%20To%20address%20this%20issue%20we%20are%20rolling%20out%20a%20hotfix.%20The%20hotfix%20rollout%20process%20across%20all%20regions%20will%20take%20approximately%2018%20hours.%20Some%20customers%20will%20continue%20to%20experience%20these%20issues%20until%20all%26nbsp%3Bregions%20have%20have%20gotten%20the%20hotfix%20applied.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F31%2020%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Sep 01 2021 09:46 AM
Updated by: