Linux Ubuntu VMs in non-responsive state for Service Map Agent- 02/25 - Resolved

[Azure-Monitor-Team]

Updated for missing links - Tuesday, 25 February 2020 17:55 UTC

The updated VM Extension has been deployed to all regions and can safely be re-enabled.

• If you removed the VM extension, as recommended earlier, you can add it back either manually or by visiting the Azure Monitor for VMs Get Started page (https://ms.portal.azure.com/#blade/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/virtualMach...). On this page you can identify the Virtual Machines that are missing the Dependency Agent extension and you can install the extension on them with a few clicks.  Note that VM Scale Sets are currently not supported on the Get Started page. You can re-enable the extension for VMSS by going to the Insights menu item from the respective VMSS blade.

• If you still have Linux Virtual Machines or VM scale sets that are running the Dependency Agent extension, you will need to take an action to upgrade them to the latest extension version. You can do this manually by removing and adding the extension, or you can use the help scripts we have put together:

For a VM download the PowerShell script from https://vminsightssupport.blob.core.windows.net/public/scripts/Manage-Extension-VM.ps1 . To update all Linux VMs in a given subscription execute in Windows PowerShell:

.\Manage-Extension-VM.ps1 -subscriptionId  <subId> -TargetOs Linux -Update

For VM scale sets download the script from https://vminsightssupport.blob.core.windows.net/public/scripts/Upgrade-VMSS-Extension.ps1 . To update a given VMSS execute:

.\Upgrade-VMSS-Extension.ps1 -SubscriptionId <subId> -ResourceGroupName <rg> -Name <vmssName> -Update

Note: these scripts may take significant amount of time to execute depending on the number of resources in your subscription/scale set.

• If you are managing the dependency agent installation manually without using a VM extension, follow these steps:

1. Download the latest Dependency Agent installer for Linux from https://aka.ms/dependencyagentlinux .
2. Check the integrity of the installer

sha256sum dependencyagentlinux 

b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

3. Check the version of the installer

sh dependencyagentlinux --version

Should report: Dependency Agent installer, version 9.10.2.9060

4. Execute the installer
   1. For an interactive installation use: sudo sh dependencyagentlinux
   2. For silent/unattended installation use sudo sh dependencyagentlinux -s

Final Update: Tuesday, 25 February 2020 05:09 UTC

We've confirmed that all systems are back to normal with no customer impact as of 02/25, 03:32 UTC. Our logs show the incident started on 02/20, 17:00 UTC and that during the 4 days 10 hours and 32 minutes that it took to resolve the issue customers using Service Map may have experienced issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.

The updated VM Extension has been deployed to all regions and can safely be re-enabled.

• If you removed the VM extension, as recommended earlier, you can add it back either manually or by visiting the Azure Monitor for VMs Get Started page. On this page you can identify the Virtual Machines that are missing the Dependency Agent extension and you can install the extension on them with a few clicks.  Note that VM Scale Sets are currently not supported on the Get Started page. You can re-enable the extension for VMSS by going to the Insights menu item from the respective VMSS blade.
• If you still have Linux Virtual Machines or VM scale sets that are running the Dependency Agent extension, you will need to take an action to upgrade them to the latest extension version. You can do this manually by removing and adding the extension, or you can use the help scripts we have put together:
• For a VM download the PowerShell script from here. To update all Linux VMs in a given subscription execute in Windows PowerShell: .\Manage-Extension-VM.ps1 -subscriptionId  <subId> -TargetOs Linux -Update
•  For VM scale sets download the script from here. To update a given VMSS execute: .\Upgrade-VMSS-Extension.ps1 -SubscriptionId <subId> -ResourceGroupName <rg> -Name <vmssName> -Update\
• Note: These scripts may take significant amount of time to execute depending on the number of resources in your subscription/scale set.
• If you are managing the dependency agent installation manually without using a VM extension, follow these steps:

                    1. Download the latest Dependency Agent installer for Linux from here.

                    2. Check the integrity of the installer

                               sha256sum dependencyagentlinux

                               b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

                   3. Check the version of the installer

                                sh dependencyagentlinux --version

                                Should report: Dependency Agent installer, version 9.10.2.9060      

                    4. Execute the installer

                        a. For an interactive installation use: sudo sh dependencyagentlinux

                        b. For silent/unattended installation use: sudo sh dependencyagentlinux -s

• Incident Timeline: 4 Days 10 Hours & 32 minutes - 02/20, 17:00 UTC through 02/25, 03:32 UTC

We understand that customers rely on Service Map as a critical service and apologize for any impact this incident caused.

-Mohini

Update: Monday, 24 February 2020 18:31 UTC

An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

• Next Update: Before 02/25 07:00 UTC

-Leela

---

Update: Sunday, 23 February 2020 19:09 UTC

An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

• Next Update: Before 02/24 19:30 UTC

-Leela

---

Update: Saturday, 22 February 2020 21:00 UTC

An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

• Next Update: Before 02/23 09:00 UTC

-Leela

---

Update: Saturday, 22 February 2020 17:00 UTC
We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.

• Work Around: 
  
  
  From VM:
  Log into VM, if unable to login, restart the VM and login.
  Run the following command as ROOT:
  dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
  Restart the VM
  From Azure Portal:
  Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
  Restart the VM.
  Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
  Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.

• Next Update: Before 02/23 05:00 UTC

-Leela

---

Update: Saturday, 22 February 2020 00:32 UTC

We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.

• Work Around: 
  
  
  From VM:
  Log into VM, if unable to login, restart the VM and login.
  Run the following command as ROOT:
  dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
  Restart the VM
  From Azure Portal:
  Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
  Restart the VM.
  Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
  Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
• Next Update: Before 02/22 07:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Leela

---

Update: Friday, 21 February 2020 10:04 UTC

Starting at 17:00 UTC on 20 Feb 2020, customers using Service Map may experience issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.

Current Workarounds:

From VM:

Log into VM, if unable to login, restart the VM and login.
Run the following command as ROOT:
dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
Restart the VM
From Azure Portal:
Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
Restart the VM.
Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.

-Jayadev

---

Initial Update: Friday, 21 February 2020 08:56 UTC

We are aware of issues within Service Map and are actively investigating. Some customers may experience issues with High CPU or non responsive behavior of VM's  which are on boarded to Service Map in multiple regions.

• Work Around: None 
• Next Update: Before 02/21 13:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Anusha

---

---