We are aware of an issue that started on 19 July 2024 at 04:09UTC, which resulted in customers experiencing unresponsiveness and startup failures on Windows machines using the CrowdStrike Falcon agent, affecting both on-premises and various cloud platforms.
CrowdStrike has released a public statement on Windows Sensor Update - crowdstrike.com addressing the matter, and it includes recommended steps for a workaround.
Customers with VMs in Azure that have the CrowdStrike Falcon agent installed have received communications via Azure Service Health (Tracking ID 1VT1-LX0). Microsoft worked with CrowdStrike to adjust the boot configuration used to start the impacted VMs and successfully download the corrected CrowdStrike signature files.
Customers can request Microsoft to apply the boot configuration by opening the Support+Troubleshooting blade in the Azure Portal for further assistance. The Subscription Owner or Contributor can provide consent by clicking the “?” in the top navigator bar of the Portal.
For environments specific to Azure, further instructions are provided below:
Recovery Options:
We have noticed that some Azure VMs are successfully updating via the CrowdStrike Falcon agent after multiple manual Virtual Machine restarts. Customers can attempt to do so as follows:
- Using the Azure Portal - attempting 'Restart' on affected VMs
- Using the Azure CLI or Azure Shell (https://shell.azure.com)
https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az-vm-restart
Please note in some cases multiple reboots may be required. Usually, the reboot of instance allows the CrowdStrike Falcon agent to update successfully. If a reboot doesn't resolve the issue, then leverage additional recovery options to resolve the issue.
Additional options for recovery:
Option 1:
We recommend customers that can restore from a backup, preferably from before 19 July 2024 at 04:09UTC, when this faulty update started rolling out.
- Customers leveraging Azure Backup can follow the following instructions: How to restore Azure VM data in Azure portal
Option 2:
Customers can attempt to remove the C-00000291*.sys file on the disk directly and potentially not need to perform detach and reattach disc.
Open Azure AZ CLI (minimum version needed is Powershell 5.1+) and run the following steps:
1.Create rescue VM with
// Creates a rescue VM, same size as the original VM in the same region. Asks for Username and password.
// Makes a copy of the OS Disk of the problem VM
// Attaches the OS Disk as Data disk to the Rescue VM
//az vm repair create -g {your-resource-group} -n {vm-name} --verbose
az vm repair create -g RGNAME -n BROKENVMNAME --verbose
**NOTE: For encrypted VM run the following command:**
az vm repair create -g RGNAME -n BROKENVMNAME --unlock-encrypted-vm --verbose
2.Then run:
// Runs the mitigation script on the Rescue VM which fixes the problem (on the os-disk copy attached as a data disk)
//az vm repair run -g {your-resource-group} -n {vm-name} --run-id win-crowdstrike-fix-bootloop --run-on-repair --verbose
az vm repair run -g RGNAME -n BROKENVMNAME --run-id win-crowdstrike-fix-bootloop --run-on-repair --verbose
3.Final step is to run:
// Removes the Fixed OS-Disk Copy from the rescue VM
// Stops the problem VM but it is not deallocated
// Attaches the fixed OS-Disk to the original VM
// Starts the original VM
// Gives prompts to delete the repair vm
//az vm repair restore -g {your-resource-group} -n {vmname} --verbose
az vm repair restore -g RGNAME -n BROKENVMNAME --verbose
Note: These steps would work for both managed and unmanaged disks. In case, if you run into capacity issues, please retry after some time.
Option 3:
Customers can attempt repairs on the OS disk by following these instructions:
Troubleshoot a Windows VM by attaching the OS disk to a repair VM through the Azure portal
Once the disk is attached, customers can attempt to delete the following file:
Windows/System32/Drivers/CrowdStrike/C-00000291*.sys
The disk can then be attached and re-attached to the original VM.
Note: We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.
Additionally, we're continuing to investigate additional mitigation options for customers and will share more information as it becomes known.
Additional recovery option references for other Windows hosted services:
- Azure Virtual Desktop (AVD): https://techcommunity.microsoft.com/t5/azure-virtual-desktop-blog/recovery-options-for-azure-virtual-desktop-session-host-vms/ba-p/4199012
- Windows endpoints: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints - Microsoft Community Hub
Microsoft
