Experiencing Data Access Issue in Azure and OMS portal for workspace data - 02/20 - Resolved

[Azure-Monitor-Team]

First published on MSDN on Feb 25, 2018

Final Update: Sunday, 25 February 2018 04:09 UTC

Hot Fix (Guest Agent - version 2.7.41491.872) has been deployed in all the regions on 2/22, 22:30 PM UTC. All of our services are running as expected and monitors are healthy. At this moment we don’t expect any issue in processing data but in case any help or info is required with respect to this issue, please reach out to Microsoft support.

• Root Cause: The failure was due to monitoring agent upgrade.
• Incident Timeline: 171 Hours  - 2/15, 19:30 UTC through 2/22, 22:30 PM UTC
  

We understand that customers rely on Azure Log Analytics service as a critical service and apologize for any impact this incident caused.

-Praveen

Update: Saturday, 24 February 2018 04:05 UTC

We continue to wait for impacted customers to get mitigated as hot-fix deployment is successfully completed for all the locations. We recommend customers to install Guest Agent (version - 2.7.41491.872) updates on VM or reboot VM who noticed their work space data missing. Regret for any inconvenience caused.

• Work Around: Rebooting the impacted VM should help in mitigating the issue
• Next Update: Before 02/25 04:30 UTC

-Praveen

Update: Friday, 23 February 2018 01:28 UTC

Hot-fix deployment is successfully completed for all the locations. We recommend customers to install Guest Agent (version - 2.7.41491.872) updates on VM or reboot VM who noticed their work space data missing. Regret for any inconvenience caused.

• Work Around: Rebooting the impacted VM should help in mitigating the issue
• Next Update: Before 02/24 01:30 UTC

-Praveen

Update: Thursday, 22 February 2018 01:14 UTC

Hot-fix deployment is completed for one of the locations. ETA for all location still remains same as 02/26. We will keep this blog updated as per the progress. Customers who are seeing their work space data missing are recommended to look for Guest Agent (version - 2.7.41491.872) updates on VM and install updates or Reboot VM to mitigate the issue.

• Work Around: Rebooting the impacted VM should help in mitigating the issue
• Next Update: Before 02/23 01:30 UTC

-Praveen

Update: Wednesday, 21 February 2018 00:15 UTC

We identified issue was due to compatibility issue with VM's running on 2.7.41491.869 and Microsoft Monitoring Agent (MMA) version 1.0.11081.3. As a resolution we have started hot-fix deployment in pre-production environment. As this issue surfaced in multiple locations so our estimated date to complete this deployment in all locations is 2/26. Customers who are seeing their work space data missing are recommended to look for Guest Agent (version - 2.7.41491.872) updates on VM and install updates or Reboot VM to mitigate the issue.

• Work Around: Rebooting the impacted VM should help in mitigating the issue
• Next Update: Before 02/22 00:30 UTC

-Praveen

Update: Tuesday, 20 February 2018 11:43 UTC

We continue to investigate issues within Azure Log Analytics service. We have identified an issue with latest monitoring agent upgrade which is causing our system to not process the data as expected. Customers using Microsoft
Monitoring Agent (MMA) extension to connect Azure Virtual Machines (VMs) to Log
Analytics who may experience the MMA extension stuck in error or transitioning
state in either the Azure Portal or the OMS Portal. On-premises VMs
which have the MMA extension installed are not impacted. The initial findings indicate that the problem began at 02/15 19:30 UTC. We currently have no estimate for resolution.

• Work Around: Rebooting the impacted VM will help in mitigating the issue
• Next Update: Before 02/21 00:00 UTC

-Mohini Nikam

Update: Tuesday, 20 February 2018 05:31 UTC

We continue to investigate issues within Azure Log Analytics service. We have identified an issue with latest monitoring agent upgrade which is causing our system to not process the data as expected. Some customers continue to experience issues while accessing workspace data in Azure portal as well as in OMS portal. The initial findings indicate that the problem began at 02/15 19:30 UTC. We currently have no estimate for resolution.

• Work Around: Rebooting the impacted VM should help in mitigating the issue
• Next Update: Before 02/20 10:00 UTC

We are working hard to resolve this issue and apologise for any inconvenience.

-Mohini Nikam

Initial Update: Tuesday, 20 February 2018 01:29 UTC

We are aware of issues within Azure Log Analytics services and are actively investigating. We have identified an issue with latest monitoring agent upgrade which is causing our system to not to process the data as expected. Some customers may experience issues while accessing workspace data in Azure portal as well as in OMS portal. We provide more updates as we learn about fixing the issue though we have workaround (reboot of VM) to mitigate this problem.

• Work Around: Rebooting of impacted VM should help in mitigating the issue
• Next Update: Before 02/20 05:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Praveen