Experiencing Issues for Azure Monitor services in West Europe - 11/07 - Resolved
Final Update: Thursday, 07 November 2019 12:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/07, 12:50 UTC. Our logs show the incident started on 11/06, 21:10 UTC and that during the 15 hours and 40 minutes that it took to resolve the issue some customers using Azure Monitor Services in West Europe may have experienced error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may have also seen Ingestion delays and latency.. Root Cause: The failure was due to azure storage outage in West Europe region. Incident Timeline: 15 Hours & 40 minutes - 11/06, 21:10 UTC through 11/07, 12:50 UTC We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused. -Mohini Update: Thursday, 07 November 2019 08:20 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 12:30 UTC -Mohini Update: Thursday, 07 November 2019 04:55 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 08:00 UTC -Mohini Initial Update: Thursday, 07 November 2019 00:49 UTC We are aware of issues within Log Analytics and are actively investigating. Some customers may experience data access issues for Log Analytics and also issues with Log alerts not being triggered as expected in West Europe region. Work Around: None Next Update: Before 11/07 03:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SindhuExperiencing Latency, Data Gap and Alerting failure for Azure Monitoring - 07/18 - Resolved
Final Update: Saturday, 18 July 2020 15:37 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/18, 11:40 UTC. Our logs show the incident started on 07/18, 07:50 UTC and that during the 3 hours 50 minutes that it took to resolve the issue some customers may have experienced Data access, Data latency, Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may have been available to be viewed with some delay in the Azure portal in multiple regions. Root Cause: The failure was due to an issue in one of our dependent services. Incident Timeline: 3 Hours & 50 minutes - 07/18, 7:50 UTC through 07/18, 11:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 18 July 2020 11:17 UTC We continue to investigate issues within Azure Monitoring services. Some customers continue to experience Data access, Data latency and Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may not be available to be viewed in the Azure portal in multiple regions. We are working to establish the start time for the issue, initial findings indicate that the problem began at 07/18 ~07:58 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 07/18 14:30 UTC -Anmol Initial Update: Saturday, 18 July 2020 08:58 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data access issues in the Azure portal, Incorrect Alert Activation, Latency and Data Loss in multiple regions. Work Around: None. Next Update: Before 07/18 11:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing latency for notifications for Azure Monitor Alerts - 01/25 - Resolved
Final Update: Saturday, 25 January 2020 12:53 UTC We've confirmed that all systems are back to normal with no customer impact as of 01/26, 04:50 UTC. Our logs show the incident started on 01/25, 05:00 UTC and that during the 23 hours and 50 minutes that it took to resolve the issue most of the customers may have experienced issues with missed or delayed Alerts notification for Azure Monitor service. Root Cause: The failure was due to one of our internal services impacted due to the SQL DB outage happened in US Govt regions. Incident Timeline: 23 Hours & 50 minutes - 01/25, 05:00 UTC through 01/26, 04:50 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 25 January 2020 10:01 UTC This issue started on 01/25 05:00 UTC and is caused by our dependency database system. Our team has been investigating this with the partner Azure database team but we do not have any root cause identified yet. Customers using Azure Monitor Services in the US Gov region continue to experience missed or delayed Alerts notifications for all type of notifications. We currently have no estimate for resolution. Work Around: Triggered alerts can viewed in Azure Portal alerts page (link). Please use the alerts page to actively monitor the health of resources till the issue is resolved. Next Update: Before 01/25 16:30 UTC -Anmol Initial Update: Saturday, 25 January 2020 07:08 UTC We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience latency for Webhook and Email notifications. Work Around: None Next Update: Before 01/25 09:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -AnmolExperiencing Alerting failure for Azure Sentinel - 09/09 - Resolved
Final Update: Wednesday, 09 September 2020 17:18 UTC We've confirmed that all systems are back to normal with no customer impact as of 09/09,16:53 UTC. Our logs show the incident started on 09/06, 07:00 UTC and that during the 3 days, 9 hours and 53 minutes that it took to resolve the issue small set of customers using Azure Sentinel and Log Search Alert may have experienced failures in running alert rules which caused alerts to not be published to the workspace. Azure Sentinel retries failed queries, so most of the queries should eventually succeed. Root Cause: The failure was due to dependency on one of the backend services. Incident Timeline: 3 Days, 9 Hours & 53 minutes - 09/06, 07:00 UTC through 09/09, 16:53 UTC We understand that customers rely on Alert rules as a critical service and apologize for any impact this incident caused. -Jayadev Initial Update: Wednesday, 09 September 2020 15:55 UTC We are aware of issues within Azure Sentinel Service and are actively investigating. Some customers may see the alert rules failing and will hence may not able to publish the alert to the workspace. Work Around: None Next Update: Before 09/09 20:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MohiniExperiencing issues in Azure Portal for Many Data Types in SUK- 09/14 - Resolved
Final Update: Tuesday, 15 September 2020 01:42 UTC We've confirmed that all systems are back to normal with no customer impact as of 9/15, 00:41 UTC. Our logs show the incident started on 9/14 13:54 UTC and that during the 10 hours and 47 minutes that it took to resolve the issue customers experienced data loss and data latency which may have resulted in false and missed alerts. Root Cause: The failure was due to a cooling failure at our data center that resulted in shutting down portions of the data center. Incident Timeline: 10 Hours & 47 minutes - 9/14 13:54 UTC through 9/15, 00:41 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Ian Update: Tuesday, 15 September 2020 01:19 UTC Root cause has been isolated to cooling failures and subsequent shutdowns in our data center which were impacting storage and our ability to access and insert data. Our infrastructure has been brought back online. We are making progress with brining the final storage devices back online. Customers should start to see signs of recover soon. Work Around: None Next Update: Before 09/15 05:30 UTC -Ian Update: Monday, 14 September 2020 20:14 UTC Starting at approximately 14:00 UTC on 14 Sep 2020, a single Zone in UK South has experienced a cooling failure. As a result, Storage, Networking and Compute resources were shut down as part of our automated processes to preserve the equipment and prevent damage. As a result the Azure Monitoring Services have experienced missed or latent data which is causing false and missed alerts. Mitigation for the cooling failure is currently in progress. An estimated time for resolution of this issue is still unknown. We apologize for the inconvenience. Work Around: None Next Update: Before 09/15 00:30 UTC -Ian Update: Monday, 14 September 2020 16:28 UTC We continue to investigate issues within Azure Monitoring Services. Root cause is related to an ongoing storage account issue. Some customers continue to experience missed or latent data which is causing false and missed alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 9/14 13:35 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 09/14 19:30 UTC -Ian Initial Update: Monday, 14 September 2020 14:44 UTC We are aware of issues within Azure Monitoring Services and are actively investigating. There is an outage on storage event in UK South which caused multiple services to be impacted. Work Around: None Next Update: Before 09/14 19:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MohiniExperiencing Alerting failure issue in Azure Portal for Many Data Types - 08/28 - Resolved
Final Update: Friday, 28 August 2020 23:39 UTC We've confirmed that all systems are back to normal with no customer impact as of 8/28, 21:30 UTC. Our logs show the incident started on 8/28, 17:30 UTC and that during the 4 hours that it took to resolve the issue, customers in the West US Region could have experience delayed or lost Diagnostic Logs. Customers using App Services Logs in Public Preview could have also experienced missed or delayed logs in all US and Canada Regions. Root Cause: The failure was due to a backend dependency. Incident Timeline: 4 Hours - 8/28, 17:30 UTC through 8/28, 21:30 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Eric SingletonExperiencing errors when accessing alerts in Azure Monitor - 01/27 - Resolved
Final Update: Wednesday, 27 January 2021 10:28 UTC We've confirmed that all systems are back to normal with no customer impact as of 01/27, 10:00 UTC. Our logs show the incident started on 01/27, 09:15 UTC and that during the 45 minutes that it took to resolve the issue some of customers may have received errors when accessing alerts. The alerts notifications were not impacted. Root Cause: We determined that a recent deployment task impacted instances of the backend service which became unhealthy, causing these errors. Incident Timeline: 45 minutes - 01/27, 09:15 UTC through 01/27, 10:00 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -AnmolExperiencing Latency and Data Loss issue in Azure Portal for Many Data Types - 11/28 - Resolved
Final Update: Saturday, 28 November 2020 15:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/28, 14:07 UTC. Our logs show the incident started on 11/27, 22:00 UTC and that during the 14 hours and 07 minutes that it took to resolve the issue some customers may have experienced delayed or missed Log Search Alerts, Latency and Data Loss in South Africa North region. Root Cause: The issue was due to power outage in South Africa North region data centers. Incident Timeline: 14 Hours & 07 minutes - 11/27, 22:00 UTC through 11/28, 14:07 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Vyom Initial Update: Saturday, 28 November 2020 05:02 UTC We are aware of issues within Application Insights and are actively investigating. Due to power outage in data center, some customers may experience delayed or missed Log Search Alerts, Latency and Data Loss in South Africa North region. Work Around: none Next Update: Before 11/28 17:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -VyomExperiencing issues viewing alerts in USGov and China region - 01/15 - Resolved
Final Update: Thursday, 16 January 2020 08:09 UTC We've confirmed that all systems are back to normal with no customer impact as of 01/16, 08:00 UTC. Our logs show the incident started on 01/11, 04:25 UTC and that during the 5 days 3 hours 35 minutes that it took to resolve the issue some of customers in US Gov and Azure China Region might have experienced issue while viewing alerts. Root Cause: The failure was due to an issue with one of our dependent service. Incident Timeline: 5 Days, 3 Hours & 35 minutes - 01/11, 04:25 UTC through 01/16, 08:00 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Monish Update: Thursday, 16 January 2020 05:32 UTC The issue got mitigated for the customers in US Gov Region as of 01/16 04:25 UTC and the engineers are continuing to work on rolling back the service Azure China for mitigating the issue. Some customer in Azure China Region might still experience issue in viewing the alerts. Work Around: None Next Update: Before 01/16 10:00 UTC -Monish Update: Thursday, 16 January 2020 02:38 UTC Root cause has been isolated to recent deployment to one of the services which was impacting alerts view in the portal. Engineers continue to work on rolling back to healthy state. Some customers may experience issues viewing alerts. Work Around: None Next Update: Before 01/16 06:00 UTC -Anupama Update: Thursday, 16 January 2020 00:15 UTC Root cause has been isolated to recent deployment to one of the services which was impacting alerts view in the portal. To address this issue engineers are working on rolling back the service to previous healthy state. Some customers may experience issues viewing alerts and we estimate 3 more hours before all alerts interface issues are addressed. Work Around: None Next Update: Before 01/16 03:30 UTC -Anupama Update: Wednesday, 15 January 2020 22:22 UTC We continue to investigate issues with alerts view in the portal. Preliminary investigation shows that a recent deployment to one of the services prevented users from viewing alerts in the portal. Customers continue to experience issues with viewing alerts in portal. We are working to establish the start time for the issue, initial findings indicate that the problem began at 01/10 ~17:30 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 01/16 00:30 UTC -Anupama Initial Update: Wednesday, 15 January 2020 20:16 UTC We are aware of issues within Azure monitor Alerts and are actively investigating. Some customers may be experiencing issues while viewing alerts in Azure portal. Work Around: none Next Update: Before 01/15 22:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -AnupamaExperiencing Data Access issue in Azure Portal for Many Data Types - 06/01 - Resolved
Final Update: Monday, 01 June 2020 17:10 UTC We've confirmed that all systems are back to normal with no customer impact as of 6/1, 16:56 UTC. Our logs show the incident started on 6/1, 13:30 UTC and that during the 3 hours and 26 minutes that it took to resolve the issue customers experienced Query and alerting failures. Root Cause: The failure was due to a backend service that became unresponsive. Incident Timeline: 3 Hours & 26 minutes - 6/1, 13:30 UTC through 6/1, 16:56 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Jeff Initial Update: Monday, 01 June 2020 16:38 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data Access issues in East US, West EU, and Central US regions. Work Around: None Next Update: Before 06/01 19:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Jeff