Blog Post

Azure Monitor Status Archive
2 MIN READ

Experiencing Data Loss for Azure Monitor Metrics - 02/03 - Resolved

Azure-Monitor-Team's avatar
Azure-Monitor-Team
Former Employee
Feb 03, 2022
Final Update: Friday, 04 February 2022 10:55 UTC

We've confirmed that all systems are back to normal with no customer impact as of 02/04, 04:00 UTC. Our logs show the incident started on 01/26, 00:00 UTC and that during the 244 hours that it took to resolve the issue, customers may have experienced delays/timeouts when viewing metrics via Azure Portal.
  • Root Cause: The failure was due to one of our backend dependency.
  • Incident Timeline:  244 Hours - 01/26, 12:00 UTC through 02/04, 04:00 UTC.
We understand that customers rely on Azure monitor as a critical service and apologize for any impact this incident caused.

-Srikanth

Update: Friday, 04 February 2022 00:26 UTC

Starting as early as 26th January 2022, some of the customers using Azure Monitor may experience delays/timeouts when viewing certain metrics via Azure Portal or any other interface. The impacted metrics includes: Disk Read Bytes/sec(Preview), Disk Read Operations/sec(Preview), Disk Write Bytes/sec(Preview), Disk Write Operations/sec(Preview). Impact is seen primarily by customers who have deployed resources against which these metrics are being queried in North America. 
Rollback is being performed in parallel across each impacted region and is expected to complete within the next 12-24 hours. As the rollback reaches each impacted region, customers whose resources leverage the impacted storage accounts may begin to see recovery.
  • Work Around: None
  • Next Update: Before 02/04 18:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Saika

Update: Thursday, 03 February 2022 19:23 UTC

Rollback to previous configuration is in progress. Some of the customers using Azure Monitor may continue to experience delays/timeouts when viewing metrics via Azure Portal or any other user interface.
  • Work Around: None
  • Next Update: Before 02/04 01:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Saika

Update: Thursday, 03 February 2022 13:21 UTC

Root cause has been isolated to recent deployment introduced a configuration error which was impacting .To address this issue we rolling back to a previous configuration Some customers may continue to experience missing Disk metrics in Azure Monitor in West US and West Central US regions.
  • Work Around: None.
  • Next Update: Before 02/03 19:30 UTC
-Srikanth

Initial Update: Thursday, 03 February 2022 07:07 UTC

We are aware of issues within Azure Monitor Metrics and are actively investigating. Some customers may experience missing Disk metrics in Azure Monitor in West US and West Central US regions.
  • Work Around: None
  • Next Update: Before 02/03 13:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Madhuri


Updated Feb 04, 2022
Version 6.0
No CommentsBe the first to comment