Experiencing Data Latency Issue in Azure portal for Azure Monitors - 12/12 - Resolved
Final Update: Thursday, 12 December 2019 08:47 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/12, 05:43 UTC. Our logs show the incident started on 12/12, 02:30 UTC and that during the 3 hours 13 minutes that it took to resolve the issue some of the customers in South East Asia Region may had experienced data access issue in the azure portal and queries from API might had failed for Log analytics, might had experienced alert failures and latency in alerting for Log Search alerts. Root Cause: The failure was due to an issue with one of our dependency service. Incident Timeline: 3 Hours & 13 minutes - 12/12, 02:30 UTC through 12/12, 05:43 UTC We understand that customers rely on Log Analytics and Log Search Alerts as a critical services and apologize for any impact this incident caused. -Monish Update: Thursday, 12 December 2019 08:11 UTC Root cause has been isolated to failure in the dependent service which was impacting the Log Analytics and Log Search Alerts clusters hosted in the South East Asia region. Some customers may experience may experience data access issue and might not receive expected alerts or may experience increased latency in receiving these alerts. Work Around: None Next Update: Before 12/12 13:30 UTC -Anusha Initial Update: Thursday, 12 December 2019 05:06 UTC We are aware of issues within Log Analytics and Log Search Alerts and are actively investigating. Some customers in south East Asia Region may experience data access issue and might not receive expected alerts or may experience increased latency in receiving these alerts. Work Around: None Next Update: Before 12/12 08:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -AnushaExperiencing Alerting failure for Azure Monitor - 12/12 - Resolved
Final Update: Thursday, 12 December 2019 13:54 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/12, 13:30 UTC. Our logs show the incident started on 12/12, 11:00 UTC and that during the 2 Hours & 30 minutes that it took to resolve the issue some of customers experienced delay in receiving all alert types. Root Cause: The failure was due to an issue with one of our dependent service. Incident Timeline: 2 Hours & 30 minutes - 12/12, 11:00 UTC through 12/12, 13:30 UTC. We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused. -AnushaExperiencing Issues for Azure Monitor services in West Europe - 11/07 - Resolved
Final Update: Thursday, 07 November 2019 12:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/07, 12:50 UTC. Our logs show the incident started on 11/06, 21:10 UTC and that during the 15 hours and 40 minutes that it took to resolve the issue some customers using Azure Monitor Services in West Europe may have experienced error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may have also seen Ingestion delays and latency.. Root Cause: The failure was due to azure storage outage in West Europe region. Incident Timeline: 15 Hours & 40 minutes - 11/06, 21:10 UTC through 11/07, 12:50 UTC We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused. -Mohini Update: Thursday, 07 November 2019 08:20 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 12:30 UTC -Mohini Update: Thursday, 07 November 2019 04:55 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 08:00 UTC -Mohini Initial Update: Thursday, 07 November 2019 00:49 UTC We are aware of issues within Log Analytics and are actively investigating. Some customers may experience data access issues for Log Analytics and also issues with Log alerts not being triggered as expected in West Europe region. Work Around: None Next Update: Before 11/07 03:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SindhuExperiencing Alerting failure and Latency in Detecting Log Alerts for Log Search Alerts - Resolved
Final Update: Thursday, 11 July 2019 05:06 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/11, 04:30 UTC. Our logs show the incident started on 07/11, 01:30 UTC and that during the 3 hours that it took to resolve the issue 80% of customers using Log alerts which are configured in East Japan region would have experienced latency in detecting log alerts and execution of Log Search queries might have timed out. Root Cause: The failure was due to issues with one of our backend services which went in to unhealthy state. Incident Timeline: 3 Hours - 07/11, 01:30 UTC through 07/11, 04:30 UTC We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused. -Rama Initial Update: Thursday, 11 July 2019 04:12 UTC We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience Alerting failures and data latency issues. Next Update: Before 07/11 07:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -LeelaExperiencing Data Access Issues for Log Analytics in West Europe - 11/15 - Resolved
Final Update: Friday, 15 November 2019 16:49 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/15, 15:56 UTC. Our logs show the incident started on 11/15, 14:20 UTC and that during 1 hour 36 minutes that it took to resolve the issue subset of customers in West Europe might have experienced data access issues while accessing data for Log Analytics and latency in Log Search Alerts. Root Cause: The failure was due to issues in one of our dependent services. Incident Timeline: 1 Hours & 36 minutes - 11/15, 14:20 UTC through 11/15, 15:56 UTC We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused. -Venkat Initial Update: Friday, 15 November 2019 15:53 UTC We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience Alerting failure in West Europe region. Work Around: None Next Update: Before 11/15 20:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing Latency, Data Gap and Alerting failure for Azure Monitoring - 07/18 - Resolved
Final Update: Saturday, 18 July 2020 15:37 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/18, 11:40 UTC. Our logs show the incident started on 07/18, 07:50 UTC and that during the 3 hours 50 minutes that it took to resolve the issue some customers may have experienced Data access, Data latency, Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may have been available to be viewed with some delay in the Azure portal in multiple regions. Root Cause: The failure was due to an issue in one of our dependent services. Incident Timeline: 3 Hours & 50 minutes - 07/18, 7:50 UTC through 07/18, 11:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 18 July 2020 11:17 UTC We continue to investigate issues within Azure Monitoring services. Some customers continue to experience Data access, Data latency and Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may not be available to be viewed in the Azure portal in multiple regions. We are working to establish the start time for the issue, initial findings indicate that the problem began at 07/18 ~07:58 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 07/18 14:30 UTC -Anmol Initial Update: Saturday, 18 July 2020 08:58 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data access issues in the Azure portal, Incorrect Alert Activation, Latency and Data Loss in multiple regions. Work Around: None. Next Update: Before 07/18 11:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperiencing Alerting failure for Log Search Alerts in West Europe region - 11/25 - Resolved
Final Update: Monday, 25 November 2019 10:01 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/25, 02:30 UTC. Our logs show the incident started on 11/25, 01:10 UTC and that during the 1 hour & 20 minutes that it took to resolve the issue some customers may not receive expected alerts or experience increased latency in receiving these alerts in West Europe region. Root Cause: The failure was due to issue with one of our backend service which became unhealthy. Incident Timeline: 1 Hour & 20 minutes - 11/25, 01:10 UTC through 11/25, 02:30 UTC We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused. -satya Update: Monday, 25 November 2019 06:45 UTC Root cause has been isolated to latency in processing messages from storage, customers might have experienced latency or failure in receiving alerts in West Europe region. To address this issue backend service resources are been scaled out. Work Around: None Next Update: Before 11/25 11:00 UTC -satya Initial Update: Monday, 25 November 2019 02:28 UTC We are aware of issues within Log Search Alerts and are actively investigating. Customers may not receive expected alerts or may experience increased latency in receiving these alerts in WEU region. Work Around: None Next Update: Before 11/25 06:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SubhashExperiencing issues with Alert Notification Service webhooks- 05/19 - Resolved
Final Update: Sunday, 19 May 2019 06:14 UTC We've confirmed that all systems are back to normal with no customer impact as of 05/19, 06:02 UTC. Our logs show the incident started on 05/18, 20:49 UTC and that during the ~9 hours 13 minutes that it took to resolve the issue some customers who have configured webhooks to send notifications may not have fired or taken action during the impacted window. Root Cause: The Failure was due to a downstream Azure Automation service which experienced issues due to configuration changes. Incident Timeline: 9 Hours & 13 minutes - 05/18, 20:49 UTC through 05/19, 06:02 UTC We understand that customers rely on Azure Notification Service as a critical service and apologize for any impact this incident caused. -Mohini Nikam Initial Update: Sunday, 19 May 2019 05:47 UTC We are aware of issues with Alert Notification with respect to configured webhooks and are actively investigating. Some customers who has configured webhook with the action items may not have receive the alerts. Work Around: None Next Update: Before 05/19 08:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Mohini NikamExperiencing Alerting failure for Log Search Alerts in West US 2- 11/13 - Resolved
Final Update: Wednesday, 13 November 2019 22:06 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/13, 21:25 UTC. Our logs show the incident started on 11/13, 20:25 UTC and that during the 1 hour that it took to resolve the issue subset of Log Analytics customers in West US 2 might have experienced failures in Log Search Alerts. Root Cause: The failure was due to issue in one of the dependent services. Incident Timeline: 1 Hour - 11/13, 20:25 UTC UTC through 11/13, 21:25 UTC We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused. -VenkatAlerts not being triggered as expected for Azure Monitor Alerts - 11/09 - Resolved
Final Update: Saturday, 09 November 2019 21:31 UTC We've confirmed that all systems are back to normal as of 11/09, 21:31 UTC. Our logs show the incident started on 11/09, 09:36 UTC and that during the 11 hours 25 minutes that it took to resolve the issue customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules). Root Cause: The failure was due to issue in one of our dependent services. Incident Timeline: 11 Hours & 55 minutes - 11/09, 09:36 UTC through 11/09, 21:31 UTC We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused. -Sindhu Update: Saturday, 09 November 2019 20:50 UTC We continue to investigate issues within Log Search Alerts, Metric Alerts, Activity Log alerts and Smart detector alerts. Root cause is not fully understood at this time. Customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules). Initial findings indicate that the problem began at 11/09 ~09:36 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 11/09 23:00 UTC -Sindhu Initial Update: Saturday, 09 November 2019 18:15 UTC We are aware of issues within Log Search Alerts, Metric Alerts, Activity Log alerts and Smart detector alerts. We are actively investigating the issue. Customers will not receive notifications for alerts which are configured to be sent through ITSM connectors (either by Action Groups or Action Rules). Work Around: None Next Update: Before 11/09 20:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -Sindhu