Experiencing failure in performing Metric Alert CRUD operations - 12/14 - Resolved
Final Update: Sunday, 15 December 2019 18:36 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/15, 18:15 UTC. Our logs show the incident started on 12/14, 06:00 UTC and that during the 1 day, 12 hours and 15 minutes that it took to resolve the issue customers could have experienced failure in performing Metric Alert CRUD operations on compute resources hosted in the East US 2 region. Root Cause: The failure was due to a dependent service. Incident Timeline: 1 day, 12 Hours & 15 minutes - 12/14, 06:00 UTC through 12/15, 18:15 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Eric Singleton Update: Saturday, 14 December 2019 14:32 UTC We continue to investigate issues within Azure Monitor. Root cause is not fully understood at this time. Some customers may continue to experience failure in performing Metric Alert CRUD operation on compute resources hosted in EastUs2 region.We are working to establish the start time for the issue, initial findings indicate that the problem began at 12/14 ~06:00 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 12/15 15:00 UTC -Anusha Initial Update: Saturday, 14 December 2019 09:59 UTC We are aware of issues within Metric Alerts and are actively investigating. Some customers may experience failure in performing Metric Alert CRUD operation on compute resources hosted in EastUs2 region. Work Around: None Next Update: Before 12/14 14:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Anusha DoddaExperiencing Alerting failure for Metric Alerts in Log Analytics Workspaces - 12/09 - Resolved
Final Update: Monday, 09 December 2019 18:43 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/09, 18:34 UTC. Our logs show the incident started on 12/09, 17:27 UTC and that during the 1 hour and 7 minutes that it took to resolve the issue customers could have experienced missing alerts in West Europe. Root Cause: The failure was due to a bad configuration Incident Timeline: 1 Hour & 7 minutes - 12/09, 17:27 UTC through 12/09, 18:34 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Eric Singleton Update: Monday, 09 December 2019 18:08 UTC Root cause has been isolated to a bad configuration which was impacting alerting. To address this issue we scaled out the deployment. Some customers may experience missing alerts in the West Europe Region. Work Around: none Next Update: Before 12/09 22:30 UTC -Eric SingletonExperiencing failure in alert creation or updation for Classic alerts - 12/18 - Resolved
Final Update: Wednesday, 18 December 2019 17:26 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/18, 16:40 UTC. Our logs show the incident started on 12/17, 17:00 UTC and that during the 23 hours 40 mins that it took to resolve the issue 90% of customers in South Central US may have received failure notifications when performing service management operations - such as create, update, delete and read for classic metric alerts hosted in this region. Root Cause: Engineers determined that a recent configuration change caused a backend service in charge of processing service requests to become unhealthy, preventing requests from completing.. Mitigation: Engineers performed a change to the service configuration to mitigate the issue. Incident Timeline: 23 Hours & 40 minutes - 12/17, 17:00 UTC through 12/18, 16:40 UTC We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -Leela Initial Update: Wednesday, 18 December 2019 16:05 UTC We are aware of issues within Classic Alerts and are actively investigating. Some customers in South Central US may experience failure in updation or creation of new alerts. Work Around: None Next Update: Before 12/18 18:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing Issues for Azure Monitor services in West Europe - 11/07 - Resolved
Final Update: Thursday, 07 November 2019 12:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/07, 12:50 UTC. Our logs show the incident started on 11/06, 21:10 UTC and that during the 15 hours and 40 minutes that it took to resolve the issue some customers using Azure Monitor Services in West Europe may have experienced error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may have also seen Ingestion delays and latency.. Root Cause: The failure was due to azure storage outage in West Europe region. Incident Timeline: 15 Hours & 40 minutes - 11/06, 21:10 UTC through 11/07, 12:50 UTC We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused. -Mohini Update: Thursday, 07 November 2019 08:20 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 12:30 UTC -Mohini Update: Thursday, 07 November 2019 04:55 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 08:00 UTC -Mohini Initial Update: Thursday, 07 November 2019 00:49 UTC We are aware of issues within Log Analytics and are actively investigating. Some customers may experience data access issues for Log Analytics and also issues with Log alerts not being triggered as expected in West Europe region. Work Around: None Next Update: Before 11/07 03:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SindhuExperiencing Alerting failure for Metric Alerts - 12/18 - Resolved
Final Update: Wednesday, 18 December 2019 15:41 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/18, 15:15 UTC. Our logs show the incident started on 12/13, 13:50 UTC and that during the 5 days 1 hours & 25 minutes that it took to resolve the issue ~45 customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events. Root Cause: The failure was due to an issue with one of our dependent service. Incident Timeline: 5 Days 1 Hours & 25 minutes - 12/13, 13:50 UTC through 12/18, 15:15 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Anmol Update: Wednesday, 18 December 2019 13:30 UTC We continue to investigate issues within Metric Alerts. Some customers in US Sovereign Cloud using Classic Alert Rules and Autoscale may continue to experience missing email notifications for alerts that were fired or autoscale events. We are working to establish the start time for the issue, initial findings indicate that the problem began at 12/13 10:00 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 12/18 17:30 UTC -Anmol Initial Update: Wednesday, 18 December 2019 10:45 UTC We are aware of issues within Metric Alerts and are actively investigating. Some customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events. Work Around: None Next Update: Before 12/18 13:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -AnmolExperiencing Latency, Data Gap and Alerting failure for Azure Monitoring - 07/18 - Resolved
Final Update: Saturday, 18 July 2020 15:37 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/18, 11:40 UTC. Our logs show the incident started on 07/18, 07:50 UTC and that during the 3 hours 50 minutes that it took to resolve the issue some customers may have experienced Data access, Data latency, Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may have been available to be viewed with some delay in the Azure portal in multiple regions. Root Cause: The failure was due to an issue in one of our dependent services. Incident Timeline: 3 Hours & 50 minutes - 07/18, 7:50 UTC through 07/18, 11:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 18 July 2020 11:17 UTC We continue to investigate issues within Azure Monitoring services. Some customers continue to experience Data access, Data latency and Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may not be available to be viewed in the Azure portal in multiple regions. We are working to establish the start time for the issue, initial findings indicate that the problem began at 07/18 ~07:58 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 07/18 14:30 UTC -Anmol Initial Update: Saturday, 18 July 2020 08:58 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data access issues in the Azure portal, Incorrect Alert Activation, Latency and Data Loss in multiple regions. Work Around: None. Next Update: Before 07/18 11:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing Alerting failure for Log Analytics Metric Alerts in East US - 11/18 - Resolved
Final Update: Tuesday, 19 November 2019 08:06 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/19, 01:13 UTC. Our logs show the incident started on 11/19, 17:00 UTC and that during the 8 hours and 13 minutes that it took to resolve the issue customers may have experienced higher than expected latency or failures regarding metric alerts during the impact window in East US region. Root Cause: The failure was due to capacity issue with one of our dependent service which created backlog. Incident Timeline: 8 Hours & 13 minutes - 11/19, 17:00 UTC through 11/19, 01:13 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Rama Update: Tuesday, 19 November 2019 02:09 UTC Root cause has been isolated to throttling of the event hub which was causing delayed ingestion. To address this issue the event hub scale limits were increased. The backlog of requests are still processing. The issue should be mitigated within 5 hours. Work Around: none Next Update: Before 11/19 07:30 UTC -Ian Cairns Initial Update: Monday, 18 November 2019 22:22 UTC We are aware of issues within Log Analytics Metric Alerts and are actively investigating. Some customers may experience Alerting failure in East US. Work Around: Next Update: Before 11/19 02:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SubhashExperiencing Alerting failure for Metric Alerts - 10/02 - Resolved
Final Update: Wednesday, 02 October 2019 17:11 UTC We've confirmed that all systems are back to normal with no customer impact as of 10/2, 16:40 UTC. Our logs show the incident started on 10/2, 01:45 UTC and that during that time customers would have experienced intermittent missing of configured metric alerts. Root Cause: The failure was due to performance bug in a backend system. Incident Timeline: 14 Hours & 55 minutes We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Jeff Update: Wednesday, 02 October 2019 15:23 UTC Root cause has been isolated to failure in one of our dependent services which was impacting Metric alerts. We continue to work on fixing the issue. Some customers in East US region may still continue to experience issues with Metric Alerts either not being delivered or may receive false positive alerts. Next Update: Before 10/02 19:30 UTC -Madhuri Update: Wednesday, 02 October 2019 09:22 UTC We continue to investigate issues within Metric Alerts. Root cause is not fully understood at this time. Some customers in East US region may still continue to experience issues with Metric Alerts either not being delivered or may receive false positive alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 10/02 01:45 UTC. Next Update: Before 10/02 15:30 UTC -Madhuri Initial Update: Wednesday, 02 October 2019 05:23 UTC We are aware of issues within Metric Alerts and are actively investigating. Some customers in East US region may experience issues with Metric Alerts either not being delivered or may receive false alerts. Next Update: Before 10/02 09:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing issues with Metric Alerts in Log Analytics for North Europe Region
Final Update: Saturday, 28 September 2019 02:19 UTC We've confirmed that all systems are back to normal with no customer impact as of 09/28, 01:50 UTC.Our logs show the incident started on 09/27,04:00 UTC and that during the 21 hours and 50 minutes that it took to resolve the issue a subset of customers in North Europe region using Log Analytics may have experienced issues with metric alerts not being delivered or may have received false positive alerts. Root Cause: The failure was due to increase on load on the dependent service. Incident Timeline: 21 Hours & 50 minutes - 09/27, 04:00 UTC through 09/28, 01:50 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Jayadev Initial Update: Saturday, 28 September 2019 01:20 UTC We are aware of issues within Metric Alerts and are actively investigating. Customers using Log Analytics in North Europe region may experience issues with metric alerts either not being delivered or may receive false alerts. Next Update: Before 09/28 05:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -JayadevIssues with Metric Alerts - 11/14 - Resolved
Final Update: Thursday, 14 November 2019 18:32 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/14, 17:40 UTC. Our logs show the incident started on 11/14, 09:00 UTC and that during 8 hours and 40 minutes that it took to resolve the issue subset of customers might have experienced issues while creating Metric Alerts during impact window. Root Cause: The failure was due to issues with one of the dependent services. Incident Timeline: 8 Hours & 40 minutes - 11/14, 09:00 UTC through 11/14, 17:40 UTC We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused. -Venkat Initial Update: Thursday, 14 November 2019 14:39 UTC We are aware of issues within Metric Alerts and are actively investigating. Some customers may experience failure in creating Metric Alerts during the impact. Work Around: None Next Update: Before 11/14 19:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Madhuri