Experiencing issues in LA and Azure Automation service in EUS2- 02/29 - Resolved

[Azure-Monitor-Team]

Final Update: Sunday, 01 March 2020 13:34 UTC

We've confirmed that all systems are back to normal with no customer impact as of 03/01, 10:30 UTC. Our logs show the incident started on 02/27, 14:00 UTC and that during the 2 days 20 hours and 30 minutes that it took to resolve the issue some customers in East US may have experienced missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may have experienced issues in solutions  - Update management and Change tracking.

• Root Cause: The failure was due to configuration changes in one of our dependent service.
• Incident Timeline: 2 Days 20 Hours & 30 minutes - 02/27, 14:00 UTC through 03/01, 10:30 UTC
  

We understand that customers rely on Azure Log Analytics and Azure Automation as critical services and apologize for any impact this incident caused.

-Mohini

---

Update: Sunday, 01 March 2020 09:34 UTC

We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions  - Update management and Change tracking.

• Work Around: There are 2 workarounds to resolve this issue:

      1) Via Azure Portal, this is for Azure Machines:

             - Login to Azure Portal. Navigate to Effected Workspace.


             - Click on "Virtual Machines" in "Workspace Data Sources" on the blade left side.

             - Select the impacted Virtual Machine. Disconnect the machine and reconnect it back. With this OMSAgent will get re-registered  and will resolve the issue


       2) Via terminal, this is for On-Prem Azure Machines:

             - Connect to the impacted linux machine.

             - Purge the installation of OMSAgent using below command. This command should be executed with root privileges.

                        sudo "/opt/microsoft/omsagent/bin/uninstall" P

             - Re-Onboard the linux machine to the Log Analytics using steps mentioned in this url (https://github.com/microsoft/OMS-Agent-for-Linux#quick-install-guide)

• Next Update: Before 03/01 22:00 UTC

-Mohini

---

Initial Update: Saturday, 29 February 2020 20:29 UTC

We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience  missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions  - Update management and Change tracking.

• Work Around: None
• Next Update: Before 03/01 10:30 UTC
• 
  

--Chandar

---

Update: Saturday, 29 February 2020 15:59 UTC

We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience  missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions  - Update management and Change tracking.

• Work Around: None
• Next Update: Before 02/29 20:00 UTC

-Mohini

---

Initial Update: Saturday, 29 February 2020 11:01 UTC

We are aware of issues within Azure Log Analytics and Azure Automation services and are actively investigating. Some customers in East US may see missing or delayed data for newly created monitoring solutions.Also Some customers whose automation accounts are in East US2 region may experience issues in solutions  - Update management and Change tracking.

• Work Around: None
• Next Update: Before 02/29 15:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Rama

---