Updated for missing links - Tuesday, 25 February 2020 17:55 UTC
The updated VM Extension has been deployed to all regions and can safely be re-enabled.
- If you removed the VM extension, as recommended earlier, you can add it back either manually or by visiting the Azure Monitor for VMs Get Started page (https://ms.portal.azure.com/#blade/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/virtualMach...). On this page you can identify the Virtual Machines that are missing the Dependency Agent extension and you can install the extension on them with a few clicks. Note that VM Scale Sets are currently not supported on the Get Started page. You can re-enable the extension for VMSS by going to the Insights menu item from the respective VMSS blade.
- If you still have Linux Virtual Machines or VM scale sets that are running the Dependency Agent extension, you will need to take an action to upgrade them to the latest extension version. You can do this manually by removing and adding the extension, or you can use the help scripts we have put together:
For a VM download the PowerShell script from https://vminsightssupport.blob.core.windows.net/public/scripts/Manage-Extension-VM.ps1 . To update all Linux VMs in a given subscription execute in Windows PowerShell:
.\Manage-Extension-VM.ps1 -subscriptionId <subId> -TargetOs Linux -Update
For VM scale sets download the script from https://vminsightssupport.blob.core.windows.net/public/scripts/Upgrade-VMSS-Extension.ps1 . To update a given VMSS execute:
.\Upgrade-VMSS-Extension.ps1 -SubscriptionId <subId> -ResourceGroupName <rg> -Name <vmssName> -Update
Note: these scripts may take significant amount of time to execute depending on the number of resources in your subscription/scale set.
- If you are managing the dependency agent installation manually without using a VM extension, follow these steps:
- Download the latest Dependency Agent installer for Linux from https://aka.ms/dependencyagentlinux .
- Check the integrity of the installer
sha256sum dependencyagentlinux
b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896 dependencyagentlinux
- Check the version of the installer
sh dependencyagentlinux --version
Should report: Dependency Agent installer, version 9.10.2.9060
- Execute the installer
- For an interactive installation use: sudo sh dependencyagentlinux
- For silent/unattended installation use sudo sh dependencyagentlinux -s
Final Update: Tuesday, 25 February 2020 05:09 UTC
We've confirmed that all systems are back to normal with no customer impact as of 02/25, 03:32 UTC. Our logs show the incident started on 02/20, 17:00 UTC and that during the 4 days 10 hours and 32 minutes that it took to resolve the issue customers using Service Map may have experienced issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.
The updated VM Extension has been deployed to all regions and can safely be re-enabled.
- If you removed the VM extension, as recommended earlier, you can add it back either manually or by visiting the Azure Monitor for VMs Get Started page. On this page you can identify the Virtual Machines that are missing the Dependency Agent extension and you can install the extension on them with a few clicks. Note that VM Scale Sets are currently not supported on the Get Started page. You can re-enable the extension for VMSS by going to the Insights menu item from the respective VMSS blade.
- If you still have Linux Virtual Machines or VM scale sets that are running the Dependency Agent extension, you will need to take an action to upgrade them to the latest extension version. You can do this manually by removing and adding the extension, or you can use the help scripts we have put together:
- For a VM download the PowerShell script from here. To update all Linux VMs in a given subscription execute in Windows PowerShell: .\Manage-Extension-VM.ps1 -subscriptionId <subId> -TargetOs Linux -Update
- For VM scale sets download the script from here. To update a given VMSS execute: .\Upgrade-VMSS-Extension.ps1 -SubscriptionId <subId> -ResourceGroupName <rg> -Name <vmssName> -Update\
- Note: These scripts may take significant amount of time to execute depending on the number of resources in your subscription/scale set.
- If you are managing the dependency agent installation manually without using a VM extension, follow these steps:
1. Download the latest Dependency Agent installer for Linux from here.
2. Check the integrity of the installer
sha256sum dependencyagentlinux
b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896 dependencyagentlinux
3. Check the version of the installer
sh dependencyagentlinux --version
Should report: Dependency Agent installer, version 9.10.2.9060
4. Execute the installer
a. For an interactive installation use: sudo sh dependencyagentlinux
b. For silent/unattended installation use: sudo sh dependencyagentlinux -s
- Incident Timeline: 4 Days 10 Hours & 32 minutes - 02/20, 17:00 UTC through 02/25, 03:32 UTC
We understand that customers rely on Service Map as a critical service and apologize for any impact this incident caused.-Mohini
Update: Monday, 24 February 2020 18:31 UTC
An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:
- If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
- If you have already uninstalled the extension, there are no actions at this time
- If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.
1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux
Execute the following in a terminal window:
wget https://aka.ms/dependencyagentlinux
2. Check the integrity of the installer
sha256sum dependencyagentlinux
b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896 dependencyagentlinux
3. Check the version of the installer
sh dependencyagentlinux --version
Should report: Dependency Agent installer, version 9.10.2.9060
4. Execute the installer
a. For an interactive installation use
sudo sh dependencyagentlinux
b. For silent/unattended installation use
sudo sh dependencyagentlinux -s
If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.
- Next Update: Before 02/25 07:00 UTC
-Leela
Update: Sunday, 23 February 2020 19:09 UTC
An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:
- If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
- If you have already uninstalled the extension, there are no actions at this time
- If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.
1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux
Execute the following in a terminal window:
wget https://aka.ms/dependencyagentlinux
2. Check the integrity of the installer
sha256sum dependencyagentlinux
b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896 dependencyagentlinux
3. Check the version of the installer
sh dependencyagentlinux --version
Should report: Dependency Agent installer, version 9.10.2.9060
4. Execute the installer
a. For an interactive installation use
sudo sh dependencyagentlinux
b. For silent/unattended installation use
sudo sh dependencyagentlinux -s
If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.
- Next Update: Before 02/24 19:30 UTC
-Leela
Update: Saturday, 22 February 2020 21:00 UTC
An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:
- If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
- If you have already uninstalled the extension, there are no actions at this time
- If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.
1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux
Execute the following in a terminal window:
wget https://aka.ms/dependencyagentlinux
2. Check the integrity of the installer
sha256sum dependencyagentlinux
b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896 dependencyagentlinux
3. Check the version of the installer
sh dependencyagentlinux --version
Should report: Dependency Agent installer, version 9.10.2.9060
4. Execute the installer
a. For an interactive installation use
sudo sh dependencyagentlinux
b. For silent/unattended installation use
sudo sh dependencyagentlinux -s
If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.
- Next Update: Before 02/23 09:00 UTC
-Leela
Update: Saturday, 22 February 2020 17:00 UTC
We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.
- Work Around:
From VM:
Log into VM, if unable to login, restart the VM and login.
Run the following command as ROOT:
dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
Restart the VM
From Azure Portal:
Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
Restart the VM.
Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
- Next Update: Before 02/23 05:00 UTC
-Leela
Update: Saturday, 22 February 2020 00:32 UTC
We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.
- Work Around:
From VM:
Log into VM, if unable to login, restart the VM and login.
Run the following command as ROOT:
dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
Restart the VM
From Azure Portal:
Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
Restart the VM.
Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
- Next Update: Before 02/22 07:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Leela
Update: Friday, 21 February 2020 10:04 UTC
Starting at 17:00 UTC on 20 Feb 2020, customers using Service Map may experience issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.
Current Workarounds:
From VM:
Log into VM, if unable to login, restart the VM and login.
Run the following command as ROOT:
dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
Restart the VM
From Azure Portal:
Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
Restart the VM.
Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
-Jayadev
Initial Update: Friday, 21 February 2020 08:56 UTC
We are aware of issues within Service Map and are actively investigating. Some customers may experience issues with High CPU or non responsive behavior of VM's which are on boarded to Service Map in multiple regions.
- Work Around: None
- Next Update: Before 02/21 13:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anusha