Experiencing Issues for Azure Monitor services in West Europe - 11/07 - Resolved
Final Update: Thursday, 07 November 2019 12:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/07, 12:50 UTC. Our logs show the incident started on 11/06, 21:10 UTC and that during the 15 hours and 40 minutes that it took to resolve the issue some customers using Azure Monitor Services in West Europe may have experienced error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may have also seen Ingestion delays and latency.. Root Cause: The failure was due to azure storage outage in West Europe region. Incident Timeline: 15 Hours & 40 minutes - 11/06, 21:10 UTC through 11/07, 12:50 UTC We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused. -Mohini Update: Thursday, 07 November 2019 08:20 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 12:30 UTC -Mohini Update: Thursday, 07 November 2019 04:55 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 08:00 UTC -Mohini Initial Update: Thursday, 07 November 2019 00:49 UTC We are aware of issues within Log Analytics and are actively investigating. Some customers may experience data access issues for Log Analytics and also issues with Log alerts not being triggered as expected in West Europe region. Work Around: None Next Update: Before 11/07 03:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SindhuExperiencing Latency, Data Gap and Alerting failure for Azure Monitoring - 07/18 - Resolved
Final Update: Saturday, 18 July 2020 15:37 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/18, 11:40 UTC. Our logs show the incident started on 07/18, 07:50 UTC and that during the 3 hours 50 minutes that it took to resolve the issue some customers may have experienced Data access, Data latency, Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may have been available to be viewed with some delay in the Azure portal in multiple regions. Root Cause: The failure was due to an issue in one of our dependent services. Incident Timeline: 3 Hours & 50 minutes - 07/18, 7:50 UTC through 07/18, 11:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 18 July 2020 11:17 UTC We continue to investigate issues within Azure Monitoring services. Some customers continue to experience Data access, Data latency and Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may not be available to be viewed in the Azure portal in multiple regions. We are working to establish the start time for the issue, initial findings indicate that the problem began at 07/18 ~07:58 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 07/18 14:30 UTC -Anmol Initial Update: Saturday, 18 July 2020 08:58 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data access issues in the Azure portal, Incorrect Alert Activation, Latency and Data Loss in multiple regions. Work Around: None. Next Update: Before 07/18 11:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing Alerting failure for Activity Log Alerts - 12/04 - Resolved
Final Update: Wednesday, 04 December 2019 13:07 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/04, 03:03 UTC. Our logs show the incident started on 12/03, 18:00 UTC and that during the 9 hours 3 minutes that it took to resolve the issue customers who have configured the Activity log alerts in US Gov Central/East Region might had experienced alerting failures. Root Cause: The failure was due to an issue with one of our dependent service. Incident Timeline: 9 Hours & 3 Minutes - 12/03, 18:00 UTC through 12/04, 03:03 UTC We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused. -MonishExperienced Management Operation failures for Activity Log Alert Rules - 10/22 - Resolved
Final Update: Thursday, 22 October 2020 09:24 UTC We've confirmed that all systems are back to normal with no customer impact as of 10/22, 08:30 UTC. Our logs show the incident started on 10/22, 01:30 UTC and that during the 7 hours that it took to resolve the issue some customers may have experienced failure notifications when accessing or performing service management operations for Azure Activity Log Alert Rules. Root Cause: The failure was due to issues with one of our dependent services. Incident Timeline: 7 Hours - 10/22, 01:30 UTC through 10/22, 08:30 UTC We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused. -SandeepExperiencing latency for notifications for Azure Monitor Alerts - 01/25 - Resolved
Final Update: Saturday, 25 January 2020 12:53 UTC We've confirmed that all systems are back to normal with no customer impact as of 01/26, 04:50 UTC. Our logs show the incident started on 01/25, 05:00 UTC and that during the 23 hours and 50 minutes that it took to resolve the issue most of the customers may have experienced issues with missed or delayed Alerts notification for Azure Monitor service. Root Cause: The failure was due to one of our internal services impacted due to the SQL DB outage happened in US Govt regions. Incident Timeline: 23 Hours & 50 minutes - 01/25, 05:00 UTC through 01/26, 04:50 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 25 January 2020 10:01 UTC This issue started on 01/25 05:00 UTC and is caused by our dependency database system. Our team has been investigating this with the partner Azure database team but we do not have any root cause identified yet. Customers using Azure Monitor Services in the US Gov region continue to experience missed or delayed Alerts notifications for all type of notifications. We currently have no estimate for resolution. Work Around: Triggered alerts can viewed in Azure Portal alerts page (link). Please use the alerts page to actively monitor the health of resources till the issue is resolved. Next Update: Before 01/25 16:30 UTC -Anmol Initial Update: Saturday, 25 January 2020 07:08 UTC We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience latency for Webhook and Email notifications. Work Around: None Next Update: Before 01/25 09:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -AnmolExperiencing Alerting failure for Azure Monitor - 08/04 - Resolved
Final Update: Tuesday, 04 August 2020 07:38 UTC We've confirmed that all systems are back to normal with no customer impact as of 08/04, 02:42 UTC. Our logs show the incident started on 08/04, 00:35 UTC and that during the 2 hours and 7 minutes that it took to resolve the issue some of the customers might have experienced delayed alerts. Alerts would have eventually fired. Root Cause: The failure was due to an issue in one of our back-end services. Incident Timeline: 2 Hours & 7 minutes - 08/04, 00:35 UTC through 08/04, 02:42 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -SaikaExperiencing Alerting failure for Activity Log Alerts - 08/17 - Resolved
Final Update: Monday, 17 August 2020 11:51 UTC We've confirmed that all systems are back to normal with no customer impact as of 08/17, 11:15 UTC. Our logs show the incident started on 08/17, 07:50 UTC and that during the 3 hours and 25 minutes that it took to resolve the issue customers using Activity Log Alerts in Azure China experienced issues with delayed alerts up to 2 hours and 50 minutes. Alerts would have eventually fired. Root Cause: The failure was due to issues with one of the backend services. Incident Timeline: 3 Hours & 25 minutes - 08/17, 07:50 UTC through 08/17, 11:15 UTC We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused. -JayadevExperiencing issues in Azure Portal for Many Data Types in SUK- 09/14 - Resolved
Final Update: Tuesday, 15 September 2020 01:42 UTC We've confirmed that all systems are back to normal with no customer impact as of 9/15, 00:41 UTC. Our logs show the incident started on 9/14 13:54 UTC and that during the 10 hours and 47 minutes that it took to resolve the issue customers experienced data loss and data latency which may have resulted in false and missed alerts. Root Cause: The failure was due to a cooling failure at our data center that resulted in shutting down portions of the data center. Incident Timeline: 10 Hours & 47 minutes - 9/14 13:54 UTC through 9/15, 00:41 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Ian Update: Tuesday, 15 September 2020 01:19 UTC Root cause has been isolated to cooling failures and subsequent shutdowns in our data center which were impacting storage and our ability to access and insert data. Our infrastructure has been brought back online. We are making progress with brining the final storage devices back online. Customers should start to see signs of recover soon. Work Around: None Next Update: Before 09/15 05:30 UTC -Ian Update: Monday, 14 September 2020 20:14 UTC Starting at approximately 14:00 UTC on 14 Sep 2020, a single Zone in UK South has experienced a cooling failure. As a result, Storage, Networking and Compute resources were shut down as part of our automated processes to preserve the equipment and prevent damage. As a result the Azure Monitoring Services have experienced missed or latent data which is causing false and missed alerts. Mitigation for the cooling failure is currently in progress. An estimated time for resolution of this issue is still unknown. We apologize for the inconvenience. Work Around: None Next Update: Before 09/15 00:30 UTC -Ian Update: Monday, 14 September 2020 16:28 UTC We continue to investigate issues within Azure Monitoring Services. Root cause is related to an ongoing storage account issue. Some customers continue to experience missed or latent data which is causing false and missed alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 9/14 13:35 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 09/14 19:30 UTC -Ian Initial Update: Monday, 14 September 2020 14:44 UTC We are aware of issues within Azure Monitoring Services and are actively investigating. There is an outage on storage event in UK South which caused multiple services to be impacted. Work Around: None Next Update: Before 09/14 19:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MohiniExperiencing Data Latency in Activity Logs - 01/29 - Resolved
Final Update: Saturday, 30 January 2021 00:45 UTC We've confirmed that all systems are back to normal with no customer impact as of 1/30, 00:30 UTC. Our logs show the incident started on 1/29, 02:00 UTC and that during the 22 hours and 30 minutes that it took to resolve the issue some customers could have experienced Data Latency with Activity Logs that are generated from Azure Resource manager as well as delayed Activity Log Alerts. Root Cause: The failure was due to a backend dependency. Incident Timeline: 22 Hours & 30 minutes - 1/29, 02:00 UTC through 1/30, 00:30 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Eric Singleton Update: Friday, 29 January 2021 22:27 UTC Root cause has been isolated to a misconfiguration from a backend dependency. To address this issue the team is currently implementing a hotfix to decrease the latency. Some customers may experience Data Latency with Activity Logs that are generated from Azure Resource Manager as well as delayed Activity Log Alerts. Work Around: None Next Update: Before 01/30 02:30 UTC -Eric Singleton Initial Update: Friday, 29 January 2021 20:36 UTC We are aware of issues within Activity Logs and are actively investigating. Some customers may experience Data Latency with Activity Logs that are generated from Azure Resource Manager. Investigation points to a start time of 1/29 02:00 UTC. Work Around: None Next Update: Before 01/29 23:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Eric SingletonExperiencing Alerting failure for alerts and action rules - 08/29 - Mitigated
Final Update: Saturday, 29 August 2020 14:15 UTC We've confirmed that all systems are back to normal with no customer impact as of 08/29, 14:05 UTC. Our logs show the incident started on 08/29, 09:15 UTC and that during the 4 hours 50 minutes that it took to resolve some of customers may experience failures accessing alerts and action rules for the resources. Alerting notifications are not impacted. Root Cause: The failure due to one of dependent service miss configuration . Incident Timeline: 4 Hours & 50 minutes - 08/29, 09:15 UTC through 08/29, 14:05 UTC We understand that customers rely on Alerts as a critical service and apologize for any impact this incident caused. -Subhash Update: Saturday, 29 August 2020 13:46 UTC We continue to investigate issues within alerting management. Some customers may experience failures accessing alerts and action rules for resources. Alerting notifications are not impacted. The problem began at 08/29 09:15 AM UTC. Work Around: None Next Update: Before 08/29 18:00 UTC -Subhash Update: Saturday, 29 August 2020 12:02 UTC We continue to investigate issues within alerting management. Some customers may experience failures accessing alerts and action rules for resources. Alerting notifications are not impacted. The problem began at 08/29 09:15 AM UTC. Work Around: None Next Update: Before 08/29 16:00 UTC -Subhash