We've confirmed that all systems are back to normal with no customer impact as of 03/01, 10:30 UTC. Our logs show the incident started on 02/27, 14:00 UTC and that during the 2 days 20 hours and 30 minutes that it took to resolve the issue some customers in East US may have experienced missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may have experienced issues in solutions - Update management and Change tracking.
- Root Cause: The failure was due to configuration changes in one of our dependent service.
- Incident Timeline: 2 Days 20 Hours & 30 minutes - 02/27, 14:00 UTC through 03/01, 10:30 UTC
-Mohini
We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions - Update management and Change tracking.
- Work Around: There are 2 workarounds to resolve this issue:
- Next Update: Before 03/01 22:00 UTC
1) Via Azure Portal, this is for Azure Machines:
- Login to Azure Portal. Navigate to Effected Workspace.
- Click on "Virtual Machines" in "Workspace Data Sources" on the blade left side.
- Select the impacted Virtual Machine. Disconnect the machine and reconnect it back. With this OMSAgent will get re-registered and will resolve the issue
2) Via terminal, this is for On-Prem Azure Machines:
- Connect to the impacted linux machine.
- Purge the installation of OMSAgent using below command. This command should be executed with root privileges.
sudo "/opt/microsoft/omsagent/bin/uninstall" P
- Re-Onboard the linux machine to the Log Analytics using steps mentioned in this url (https://github.com/microsoft/OMS-Agent-for-Linux#quick-install-guide)
We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions - Update management and Change tracking.
- Work Around: None
- Next Update: Before 03/01 10:30 UTC
We continue to investigate issues within Log Analytics and Azure Automation services. Some customers in East US continue to experience missing or delayed data for newly created monitoring solutions. Also Some customers whose automation accounts are in East US2 region may experience issues in solutions - Update management and Change tracking.
- Work Around: None
- Next Update: Before 02/29 20:00 UTC
We are aware of issues within Azure Log Analytics and Azure Automation services and are actively investigating. Some customers in East US may see missing or delayed data for newly created monitoring solutions.Also Some customers whose automation accounts are in East US2 region may experience issues in solutions - Update management and Change tracking.
- Work Around: None
- Next Update: Before 02/29 15:30 UTC
-Rama
