Experiencing Latency and Data Loss issue in Azure Portal for Many Data Types - 12/14 - Resolved
Final Update: Saturday, 14 December 2019 17:39 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/14, 17:00 UTC. Our logs show the incident started on 12/14, 16:26 UTC and that during the 34 minutes that it took to resolve the issue customers ingesting telemetry in Australia South East geographical region may have experienced intermittent data latency, data gaps and incorrect alert activation Root Cause: The failure was due to one of our backend services Incident Timeline: 34 minutes - 12/14, 16:26 UTC through 12/14, 17:00 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Eric SingletonExperiencing Data Latency issue in Azure Portal for Many Data Types - 12/12 - Resolved
Final Update: Thursday, 12 December 2019 18:54 UTC We've confirmed that all systems are back to normal with no customer impact as of 12/12, 16:40 UTC. Our logs show the incident started on 12/12, 16:15 UTC and that during the 25 minutes that it took to resolve the issue customers could have experience data latency in the West US 2 region. Root Cause: The failure was due to one of our backend services. Incident Timeline: 25 minutes - 12/12, 16:15 UTC through 12/12, 16:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Eric SingletonIssues with .Net Core web applications in App Services monitored with Application Insights - 11/12
Final Update: Tuesday, 03 December 2019 22:48 UTC We've confirmed that all systems are back to normal and the customers using .NET Core web applications monitored with Application Insights hosted in App Services no longer experience exceptions. The fix was rolled out across all regions. We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Saika ____________________________________________________________________________________________________________________________________________________________ Update: Tuesday, 12 November 2019 20:50 UTC Customers using .NET Core web applications monitored with Application Insights hosted in App Services in rare cases might experience crashes with following exception: "System.IO.FileNotFoundException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=4.0.4.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified." Such exception can happen when .NET Core web application has a dependency on 'System.Diagnostics.DiagnosticSource%u2019 binary and has this binary file deployed to %u2018wwwroot%u2019 folder, which is rare. We are aware of the issue and are rolling out the fix across different regions. Work Around: Please set previous version of monitoring extension by changing this app setting if you experience the above issue: ApplicationInsightsAgent_EXTENSIONVERSION=2.8.24 We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -VenkatExperiencing Issues for Azure Monitor services in West Europe - 11/07 - Resolved
Final Update: Thursday, 07 November 2019 12:29 UTC We've confirmed that all systems are back to normal with no customer impact as of 11/07, 12:50 UTC. Our logs show the incident started on 11/06, 21:10 UTC and that during the 15 hours and 40 minutes that it took to resolve the issue some customers using Azure Monitor Services in West Europe may have experienced error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may have also seen Ingestion delays and latency.. Root Cause: The failure was due to azure storage outage in West Europe region. Incident Timeline: 15 Hours & 40 minutes - 11/06, 21:10 UTC through 11/07, 12:50 UTC We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused. -Mohini Update: Thursday, 07 November 2019 08:20 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 12:30 UTC -Mohini Update: Thursday, 07 November 2019 04:55 UTC We continue to investigate issues within Azure monitor services. This issue started since 11/06/2019 21.10 UTC and is caused by our dependency storage system. Our team has been investigating this with partner Azure storage team but we do not have any root cause identified yet. Customers using Azure Monitor Services in West Europe may experience error while querying or/and ingesting data along with alerts failures or latent alerts. Customers who are using Service Map in West Europe may also experience Ingestion delays and latency. We provide an update as we learn. Work Around: None Next Update: Before 11/07 08:00 UTC -Mohini Initial Update: Thursday, 07 November 2019 00:49 UTC We are aware of issues within Log Analytics and are actively investigating. Some customers may experience data access issues for Log Analytics and also issues with Log alerts not being triggered as expected in West Europe region. Work Around: None Next Update: Before 11/07 03:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SindhuExperiencing Data Latency issue in Azure Portal for Many Data Types - 09/03 - Resolved
Final Update: Thursday, 03 September 2020 22:36 UTC We've confirmed that all systems are back to normal with no customer impact as of 09/03, 21:56 UTC. Our logs show the incident started on 09/03, 09:30 UTC and that during the 12 hours and 26 minutes that it took to resolve the issue some customers might have experienced intermittent data latency, data gaps and incorrect alert activation in West Europe region. Root Cause: The issue was due to hardware failure in one of the region's data centers affecting the network. Incident Timeline: 12 Hours & 26 minutes - 09/03, 09:30 UTC through 09/03, 21:56 UTC We understand that customers rely on Application Insights and Azure Log Analytics as critical services and apologize for any impact this incident caused. -Saika Update: Thursday, 03 September 2020 20:00 UTC Root cause has been isolated to a hardware failure in one of the region's data centers affecting the network and systems are still catching up to get back to normal. Some customers may still continue to experience intermittent data latency, data gaps and incorrect alert activation in West Europe region. Next Update: Before 09/04 00:00 UTC -Saika Update: Thursday, 03 September 2020 15:30 UTC We continue to investigate issues within Application Insights and Azure Log Analytics. Some customers continue to experience intermittent data latency, data gaps and incorrect alert activation in West Europe region. The initial findings suggest that the issue began at 09/03 ~09:30 UTC. Work Around: None Next Update: Before 09/03 20:30 UTC -Sandeep Update: Thursday, 03 September 2020 12:38 UTC We continue to investigate issues within Application Insights and Azure Log Analytics in West Europe region. Root cause is not fully understood at this time. Some customers continue to experience intermittent data latency, data gaps and incorrect alert activation. We are working to establish the start time for the issue, initial findings indicate that the problem began at 09/03 ~09:30 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 09/03 15:00 UTC -Rama Initial Update: Thursday, 03 September 2020 10:34 UTC We are aware of issues with Data Latency within Application Insights and Log Analytics in West Europe region and are actively investigating. Some customers may experience intermittent data latency, data gaps and incorrect alert activation. Work Around: None Next Update: Before 09/03 14:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -SandeepSmart Detection Alert mails sent for failure anomalies were malformed - 03/27 - Resolved
Final Update: Wednesday, 27 March 2019 09:41 UTC We've confirmed that all systems are back to normal with no customer impact as of 03/25/2019 21:15 UTC. Our logs show the incident started on 03/27/2019 09:15 UTC and that during the 36 hours that it took to resolve the issue some customers may have received malformed failure anomalies alert mails. The metric and availability alert emails were not impacted. Root Cause: The failure was due to a recent change in our mailing service. Incident Timeline: 36 Hours - 03/25/2019 21:15 UTC through 03/27/2019 09:15 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Varun Initial Update: Wednesday, 27 March 2019 09:02 UTC We are aware of issues within Application Insights and are actively investigating. Some customers may have received malformed failure anomalies alert mails. The links in the alert mails still work correctly. The metric and availability alert emails were not impacted. Work Around: As a workaround, Customers who received the malformed emails can directly verify the details on the Azure Portal. Next Update: Before 03/27 11:30 UTC We are working hard to resolve this issue and apologize for any inconvenience. -VarunExperiencing Latency, Data Gap and Alerting failure for Azure Monitoring - 07/18 - Resolved
Final Update: Saturday, 18 July 2020 15:37 UTC We've confirmed that all systems are back to normal with no customer impact as of 07/18, 11:40 UTC. Our logs show the incident started on 07/18, 07:50 UTC and that during the 3 hours 50 minutes that it took to resolve the issue some customers may have experienced Data access, Data latency, Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may have been available to be viewed with some delay in the Azure portal in multiple regions. Root Cause: The failure was due to an issue in one of our dependent services. Incident Timeline: 3 Hours & 50 minutes - 07/18, 7:50 UTC through 07/18, 11:40 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Anmol Update: Saturday, 18 July 2020 11:17 UTC We continue to investigate issues within Azure Monitoring services. Some customers continue to experience Data access, Data latency and Data Loss, incorrect Alert activation, missed or delayed Alerts and Azure Alerts created during the impact duration may not be available to be viewed in the Azure portal in multiple regions. We are working to establish the start time for the issue, initial findings indicate that the problem began at 07/18 ~07:58 UTC. We currently have no estimate for resolution. Work Around: None Next Update: Before 07/18 14:30 UTC -Anmol Initial Update: Saturday, 18 July 2020 08:58 UTC We are aware of issues within Application Insights and Log Analytics and are actively investigating. Some customers may experience Data access issues in the Azure portal, Incorrect Alert Activation, Latency and Data Loss in multiple regions. Work Around: None. Next Update: Before 07/18 11:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -MadhuriExperienced Latency and Data Gaps in East US2 region - 10/24 - Resolved
Final Update: Saturday, 24 October 2020 14:43 UTC We've confirmed that all systems are back to normal with no customer impact as of 10/24, 14:15 UTC. Our logs show the incident started on 10/23, 19:01 UTC and that during the 19 hour 14 minutes that it took to resolve the issue some customers may have experienced data latency and data gaps in East US2 region. Root Cause: The failure was due to bad configuration in our dependent services. Incident Timeline: 19 Hours - 10/23, 19:01 UTC through 10/24, 14:15 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Sandeep Update: Saturday, 24 October 2020 08:04 UTC Root cause has been isolated to a backend infrastructure component that was accidentally deleted. We have recovered the component and waiting for complete propagation of the restored configuration. Customers may continue to experience data gaps and latency in East US2 region. Work Around: None Next Update: Before 10/24 20:30 UTC -Sandeep Update: Saturday, 24 October 2020 04:24 UTC Root cause has been isolated to a backend infrastructure component that was accidentally deleted. We have recovered the component and waiting for complete propagation of the restored configuration. Work Around: None Next Update: Before 10/24 08:30 UTC -Sandeep Update: Saturday, 24 October 2020 03:18 UTC Root cause has been isolated to a backend infrastructure component that was accidentally deleted. We have recovered the component and waiting for complete propagation of the restored configuration. Work Around: Next Update: Before 10/24 06:30 UTC -Chandar Update: Saturday, 24 October 2020 02:55 UTC We continue to investigate issues that impacts Workspace-enabled Application Insights resource only. Root cause is not fully understood at this time. Customers might experience data gaps and latency. We are working to establish the start time for the issue, initial findings indicate that the problem began at <10/23 ~19:05 UTC>. Work Around: Next Update: Before 10/24 05:00 UTC -Chandar Initial Update: Saturday, 24 October 2020 02:51 UTC We are aware of issues within Application Insights and are actively investigating. Some customers may experience Latency and Data Loss. Work Around: Next Update: Before 10/24 05:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -ChandarExperiencing Data Access issue in Application Insights - 10/30 - Resolved
Final Update: Wednesday, 30 October 2019 15:32 UTC We've confirmed that all systems are back to normal with no customer impact as of 10/30, 13:52 UTC. Our logs show the incident started on 10/30, 10:17 UTC and that during the 3 hours & 35 minutes that it took to resolve the issue some of customers might have experienced issues with Data Access and connecting to resources hosted in West Europe region. Root Cause: The failure was due to an issue in one of our backend service. Incident Timeline: 3 Hours & 35 minutes - 10/30, 10:17 UTC through 10/30, 13:52 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -Naresh Initial Update: Wednesday, 30 October 2019 13:00 UTC We are aware of issues within Application Insights and are actively investigating. Some customers may experience issues with Data Access in West Europe region. Work Around: None Next Update: Before 10/30 17:00 UTC We are working hard to resolve this issue and apologize for any inconvenience. -NareshExperiencing Webtest activation Notificaiton failure issue in Azure Portal - 06/06 - Resolved
Final Update: Thursday, 06 June 2019 18:41 UTC We've confirmed that all systems are back to normal with no customer impact as of June 6 2019, 18:32 UTC. Our logs show the incident started on June 6,2019 08:05 UTC and during impact timeline that it took to resolve the issue 100% of customers who subscribed to Webtest Alert Activation notification would have missed receiving their activation notification emails. Root Cause: The failure was due to downstream notification service. Lessons Learned: We are looking to the telemetry for improvements on our ability to detect such issues sooner. Incident Timeline: 8 Hours & 27 minutes - June 6,2019 08:05 UTC through June 6 2019, 18:32 UTC We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused. -chandar Update: Thursday, 06 June 2019 17:42 UTC Root cause has been isolated to a template in one of the downstream notification service which was impacting Webtest alert mail activated notifications. We are deploying a hotfix to address this issue . Gen2 alerts are working as expected. We estimate 2 hours before all impacts are addressed. Work Around: None Next Update: Before 06/06 20:00 UTC -chandar