Blog Post

Intune Customer Success
12 MIN READ

New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

Intune_Support_Team's avatar
Intune_Support_Team
Icon for Microsoft rankMicrosoft
Jul 20, 2024

The Microsoft Recovery Tool was updated 7/22/2024 as version 3.1. While fundamentally there are no functional changes to the tool, for the Recover from WinPE option, we have expanded the logging, reattempt logic, and error handling. For the Recover from safe mode for USB delivery, we’ve added in better user awareness cues for when to run the repair command.

 

7/23/2024: Microsoft notes that CrowdStrike has updated its Remediation and Guidance Hub: Falcon Content Updates for Windows Hosts.

 

As a follow-up to the CrowdStrike Falcon agent issue impacting Windows clients and servers, Microsoft has released an updated recovery tool with two repair options to help IT admins expedite the repair process. The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386. In this post, we include detailed recovery steps for Windows client, servers, and OS's hosted on Hyper-V. The two repair options are as follows:

  • Recover from WinPE – this option produces boot media that will help facilitate the device repair.
  • Recover from safe mode – this option produces boot media so impacted devices can boot into safe mode. The user can then login using an account with local admin privileges and run the remediation steps.

 

Determining which option to use

 

Recover from WinPE (recommended option)
This option quickly and directly recovers systems and does not require local admin privileges. However, you may need to manually enter the BitLocker recovery key (if BitLocker is used on the device) and then repair impacted systems. If you use a third-party disk encryption solution, please refer to vendor guidance to determine options to recover the drive so that the remediation script can be run from WinPE.

 

Recover from safe mode
This option may enable recovery on BitLocker-enabled devices without requiring the entry of BitLocker recovery keys. For this option, you must have access to an account with local administrator rights on the device. Use this approach for devices using TPM-only protectors, devices that are not encrypted, or situations where the BitLocker recovery key is unknown. However, if utilizing TPM+PIN BitLocker protectors, the user will either need to enter the PIN if known, or the BitLocker recovery key must be used. If BitLocker is not enabled, then the user will only need to sign in with an account with local administrator rights. If third-party disk encryption solutions are utilized, please work with those vendors to determine options to recover the drive so the remediation script can be run.


Additional considerations
Although the USB option is preferred, some devices may not support USB connections. In such cases, we provide detailed steps below for using the Preboot Execution Environment (PXE) option. If the device cannot connect to a PXE network and USB is not an option, reimaging the device might be a solution.

 

As with any recovery option, test on multiple devices prior to using it broadly in your environment.

 

Prerequisites to create the boot media

  1. A Windows 64-bit client with at least 8GB of free space from which the tool can be run to create the bootable USB drive.
  2. Administrative privileges on the Windows client from prerequisite #1.
  3. A USB drive with min 1GB and max of 32GB. All existing data on this USB will be wiped and will be formatted automatically to FAT32.

 

Instructions to generate the WinPE recovery media
To create recovery media, follow these steps on the 64-bit Windows client mentioned in prerequisite #1:

  1. Download the signed Microsoft Recovery Tool from the Microsoft Download Center.
  2. Extract the PowerShell script from the downloaded solution.
  3. Run MsftRecoveryToolForCSv31.ps1 from an elevated PowerShell prompt.
  4. The ADK will download and media creation will start. It may take several minutes to complete.
  5. Choose one of the two options mentioned above for recovering affected devices (see additional details below).
  6. Optionally select a directory that contains driver files to import into the recovery image. Keyboard and mass storage drivers may be needed. Network or other drivers are not required. We recommend you select “N” to skip this step. The tool will import any SYS and INI recursively under the specified directory.
  7. Select the option to either generate an ISO or USB drive and specify drive letter.

NOTE: Microsoft recommends uninstalling the ADK after completing the remediation process. If the ADK is needed for other tasks on the media creation machine, it should be uninstalled and then reinstalled to avoid any remnants from the remediation tool.

 

Prerequisites for using the boot media
The BitLocker recovery key for each BitLocker-enabled impacted device on which the recover media is used may be required. If you are using TPM-only protectors and using the safe boot option, then the recovery key will not be required. If you are using TPM+PIN protectors, then you may need the recovery key if you do not know the PIN for the device.

 

Using Recovery from WinPE media

  1. Insert the USB key into an impacted device.
  2. Reboot the device.
  3. During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
  4. From the BIOS boot menu, choose Boot from USB and continue.
  5. The tool will run.
  6. If BitLocker is enabled, the user will be prompted for the BitLocker recovery key including the dashes. The recovery key options are provided here. For third-party device encryption solutions, follow any steps provided by the vendor to gain access to the drive.
  7. The tool will run the issue-remediation scripts as recommended by CrowdStrike.
  8. Once complete, remove the USB drive and reboot the device normally.

 

Using Safe Boot media
To repair an impacted device without using the BitLocker recovery key and if you have access to the local administrator account:

  1. Insert the USB key into an impacted device.
  2. Reboot the device.
  3. During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
  4. From the BIOS boot menu, choose Boot from USB and continue.
  5. The tool runs.
  6. The following message appears: "This tool will configure this machine to boot in safe mode. WARNING: In some cases you may need to enter a BitLocker recovery key after running."
  7. Press any key to continue.
  8. The following message appears: "Your PC is configured to boot to Safe Mode now."
  9. Press any key to continue.
  10. The machine reboots into safe mode.
  11. The user runs repair.cmd from the root of the media/USB drive. The script will run the remediation steps as recommended by CrowdStrike.
  12. The following message appears: "This tool will remove impacted files and restore normal boot configuration. WARNING: You may need BitLocker recovery key in some cases. WARNING: This script must be run in an elevated command prompt."
  13. Press any key to continue.
  14. The user repair will run and the normal boot flow will be restored.
  15. Once successful, the user will see the following message: “Success. System will now reboot.”
  16. Press any key to continue. The device will reboot normally.

 

Using recovery media on Hyper-V virtual machines
The recovery media can be used to remediate impacted Hyper-V virtual machines. To do so, select the option to generate an ISO when creating the recovery media using the steps above. For non-Hyper-V virtual machines, follow instructions provided by your hypervisor vendor to utilize the recovery media.

 

Steps to Recover Hyper-V virtual machines

  1. On an impacted virtual machine, add a DVD Drive under Hyper-V settings > SCSI Controller.

Screenshot for where to add the DVD Drive.

 

  1. Browse to the recovery ISO and add it as an Image file under Hyper-V Settings > SCSI Controller > DVD Drive.

 

Screenshot of where to add the image file.

 

  1. Note the current Boot order so that it can be restored back manually later.

 

Screen shot of the original boot order.

 

  1. Change the Boot order to move the added DVD Drive the first boot entry.

 

Screenshot of the change to the boot order.

 

  1. Start the virtual machine and select any key on keyboard to continue booting to the ISO image.
  2. Depending on whether the option to use WinPE or safe mode was used when creating the recovery media, follow the steps above to repair the system.
  3. Set the boot order back to the original boot settings from the virtual machine’s Hyper-V settings.
  4. Reboot normally.

 

Using PXE for Recovery
For most customers, the options listed above or following the steps in the KBs linked towards the end of this post will help restore your devices. However, if devices are unable to use the option to recover from USB, for example, because of security policies or port availability, IT admins can use PXE to remediate.


To use this solution, you can use the Windows Imaging Format (WIM) that the Microsoft Recovery Tool creates in an existing PXE environment as long as the impacted devices are on the same subnet as the PXE server. Alternatively, you can either use the PXE server approach outline below. This option works best when the PXE server can be moved subnet to subnet easily for remediation purposes.

 

Prerequisites for PXE Recovery

  1. An x64 machine (referred to as the “PXE server”) which will host the boot image.
    1. The PXE server can run on any supported Windows client x64 operating system.
    2. The PXE server should have network access to download the tools either from https://go.microsoft.com/fwlink/?linkid=2281008 or from an internal link on your network.
    3. The PXE server should have inbound firewall rules created for UDP ports 67, 68, 69, 547, and 4011. The PXE tool downloaded (MSFTPXEToolForCS.exe) will update the Windows Firewall settings on the PXE Server. If using a third-party firewall, create rules following their recommendations.
      NOTE: This script does not clean up the firewall rules. You should remove these firewall rules after remediation is complete. You can run MSFTPXEToolForCSv31.ps1 Clean from an elevated PowerShell prompt to remove these rules from the Windows firewall.
    4. You'll need admin privileges to run the PXE tool.
    5. PXE server requires the VC Redistributable. The latest version can be downloaded and installed from: https://aka.ms/vs/17/release/vc_redist.x64.exe
  2. The affected Windows devices should be on the same subnet as the PXE Server and should be hard-wired instead of using a Wi-Fi network.

 

Configuring the PXE server

  1. Download the package from https://go.microsoft.com/fwlink/?linkid=2281008.
    1. The zip file contains all the files needed. Extract the contents of the zip to any directory.
  2. From an elevated PowerShell prompt, change to the directory where you extracted the files and execute the following from an elevated PowerShell prompt: MSFTPXEToolForCSv31.ps1
    1. The script will initiate a scan for ADK and ADK WinPE Add-On installation on the machine and install them if missing. Accept the on-screen license request to proceed with installation.
    2. The script will generate the remediation scripts and create a valid boot image.
    3. If required, accept the prompt and provide a path containing the driver files. Driver files may be required for keyboard and/or mass storage. Generally adding drivers will not be required. If no additional driver files are needed, select ‘n’.
    4. You will be given the option to set up the PXE Server to deliver a default remediation image or a safe mode image with the following prompts:
      1. “1. Boot to WinPE to remediate the issue. It requires entering BitLocker recovery key if system disk is BitLocker encrypted.”
      2. “2. Boot to WinPE configure safe mode and run repair command after entering safe mode. This option is less likely to require BitLocker recovery key if system disk is BitLocker encrypted.”
    5. The script will generate the required distribution files and provide the path where the PXE server tool is copied.
  3. Ensure that the prerequisite, https://aka.ms/vs/17/release/vc_redist.x64.exe, is installed and all the prerequisites are met.
  4. From an elevated command prompt, change to the directory where the PXE server tool is copied. Run .\MSFTPXEToolForCS.exe to launch the listener process.
    1. You will not receive additional responses as this is the PXE server handling connections. Do not close this window as it will stop the PXE server.
    2. You should monitor the PXE server progress in the MSFTPXEToolForCS.log file within the directory.
      NOTE: If you would like to run multiple PXE servers for different subnets you will need to copy the directory where the PXE server tool is copied and execute steps 3 & 4 above. In addition, Microsoft recommends uninstalling the ADK after completing the remediation process. If the ADK is needed for other tasks on the media creation machine, it should be uninstalled and then reinstalled to avoid any remnants from the remediation tool.
    3. Additional references:

 

Recover an impacted device

  1. The impacted device must be on the same subnet as PXE Server.
    1. If the devices are in a different subnet, configure IP Helpers in your network environment to enable the discovery of the PXE server.
  2. If the impacted device is not configured for PXE boot, follow these steps:
    1. On the impacted device, enter the BIOS\UEFI
    2. This operation is different across different models and manufacturers. Refer to documentation provided by the manufacturer for your machine (make and model).
    3. Common options for accessing the BIOS\UEFI involve inputting a key like F2, F12, DEL, or ESC during the startup.
    4. Ensure Network boot is enabled on the device.
    5. Refer to documentation from your manufacturer for additional guidance
      1. Reference documentation: How to Enable Pre-Boot Execution Environment (PXE Boot) in BIOS?
    6. Configure the network boot option as the first boot priority.
    7. Save the new settings and reboot the client device for the settings to apply.
    8. Device will now boot from PXE.
  3. PXE boot the affected machine
    1. Depending on whether you chose to create WinPE or safe mode media, the user will either be prompted to boot to Windows PE and the remediation script will execute automatically, or they will boot to safe mode where the user needs to login with local admin credentials and execute the script manually. See the detailed steps above for the experience for both WinPE and safe mode.
      1. If you created the safe mode option, you will need to run the following commands as admin, from an elevated Command prompt
        a. del %SystemRoot%\System32\drivers\CrowdStrike\C-00000291*.sys
        b. bcdedit /deletevalue {current} safeboot
        c. shutdown -r -t 00
  4. Once complete, reboot the device normally by responding to the prompt on the screen. Enter the BIOS\UEFI and update the boot order manually to remove PXE boot if needed.

 

For more information on the issue impacting Windows clients and servers running the CrowdStrike Falcon agent, please see:


Thank you for your continued feedback through this post, support, and other feedback channels, such as @IntuneSuppTeam on X. Please note this tool does not use Microsoft Intune, but we're sharing as a Support tip to help customers. We’ll continue to provide updates to this post as needed.

Updates to this post
[7/21/2024] - Complete update of the blog post to describe the two options for recovery now using the updated signed Microsoft Recovery Tool. Many customers have used the tool and provided feedback, which we have incorporated, thank you. The new release includes a new option for recovery using safe boot, the option to generate ISO or USB, a fix for ADK detection when the Windows Driver Kit is installed, and a fix for the USB disk size check.

[7/21/2024] - PXE recovery option added. 

[7/22/2024] - Signed Microsoft Recovery Tool updated (multiple changes summarized below). Also updated a Windows KB article link.

[7/23/2024] - Minor text updates.

[7/31/2024] - Updated to clarify that the existing ADK should be uninstalled and then reinstalled after remediation to remove any remnants.

 

Changes to Recover from WinPE for USB, PXE and ISO delivery:

  • Additional logging for the commands that get run to help with troubleshooting.
  • Added the capability to try again or change the drive if printing BitLocker information or unlocking the drive does not process successfully as expected.

Screenshot of remediation script to try another drive

  • In the event that you wish to recover a drive that is not encrypted with BitLocker, select C to continue when prompted.
  • To select a different drive to re-attempt BitLocker recovery, select T to Try again when prompted.
  • Depending on configuration, a different message may appear. When prompted with “Error: No key protectors found”, hit enter to return no recovery key and to continue with remediation:

 

Configuration screen share.

 

  • Error handling improvements based on customer feedback.

 

Change to Recover from safe mode for USB and ISO delivery

  • Better user awareness of cues of when to run Repair.cmd

 

The zip and PS1 names have been updated to:

  • MsftRecoveryToolForCSv2.ps1 -> MsftRecoveryToolForCSv31.ps1
  • MSFTPXEInitToolForCS.ps1 -> MSFTPXEToolForCSv31.ps1 - Init removed from the file name
Updated Aug 01, 2024
Version 14.0
Support Tip
windows
  • BruceK825's avatar
    BruceK825
    Copper Contributor
    Jul 21, 2024

    RobYork  thanks, no worries.   I tried to handle it in the code below, in case it helps anyone.   For now, we're trying to detect Bitlocker encryption on or off based on the "manage-bde -status" command and parsing output looking for the "Lock Status:"   If its "Locked" I then parse the "Volume " line of the output to get to the drive letter.  If there's multiple drive letters this won't work well.  Maybe there's a better way, since we found the manage-bde -status output to be limited in the Windows PE environment with encryption enabled.   

     

    If the target computer isn't using Bitlocker encryption, the batch file loops through drive letters A-Z and finds the one where CrowdStrike exists.  That seems to work pretty consistently.   This isn't fool-proof but hopefully someone with better .bat file skills could enhance it.    Thanks, Bruce  

     

    7/22:  I haven't done .bat files in decades a colleague noticed I needed a "setlocal" at the beginning to clear variables.  Also, the parsing of "manage-bde /status" doesn't seem to work in the WinPE environment so this approach is flawed.  

     

    setlocal
cls
@echo off
for /f "delims=" %%a in ('manage-bde -status') do ECHO.%%a | FIND /I "Lock Status:          Locked" > Nul &&  GoTo BitLockerOn
GoTo BitLockerOff

:BitLockerOn
for /f "delims=" %%a in ('manage-bde -status') do ECHO.%%a | FIND /I "Volume " > Nul && for /f "tokens=2" %%i in ("%%a") do set word2=%%i && set drive=%word2%
echo Using drive %drive%
echo If your device is BitLocker encrypted use your phone to log on to https://aka.ms/aadrecoverykey. Log on with your Email ID and domain account password to find the BitLocker recovery key associated with your device.
echo.
manage-bde -protectors %drive% -get -Type RecoveryPassword
echo.
set /p reckey="Enter recovery key for this drive if required: "
IF NOT [%reckey%] == [] (
	echo Unlocking drive %drive%
	manage-bde -unlock %drive% -recoverypassword %reckey%
)
Goto DeleteLogic


:BitLockerOff
for %%D in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do if exist "%%D:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"  (
	set drive=%%D:
	echo Using drive %drive%
	echo.
	Goto DeleteLogic
)

:DeleteLogic
	del %drive%\Windows\System32\drivers\CrowdStrike\C-00000291*.sys
	echo Done performing cleanup operation.

     

     

  • aarony's avatar
    aarony
    Brass Contributor
    Jul 21, 2024

    RobYork Confirmed a machine where NO kits are installed works.  If you have the "Windows Driver Kit" installed, you get a false positive for the ADK in your .ps1 script.  It's because the "Windows Driver Kit" installs the DISM components into the '\Assessment and Deployment Kit\Deployment Tools\AMD64' folder, which your .ps1 script is using to detect if you have ADK.

  • stephc_msft's avatar
    stephc_msft
    Icon for Microsoft rankMicrosoft
    Jul 21, 2024

    Just a thought/idea as I was thinking of ideas for a similar tool.
    One of the big issues for enterprises with thousands of systems, each with individual BitLocker recovery keys, is the slow one-at-a-time acquiring and entering of the key.
    I wonder if we could somehow bulk download a database of keys, each with their Identifier and the Recovery Key, or the .BEK files, (from the Enterprise based storage [somehow]), and then have the usb based tool search and find and use the key from that database.
    Obviously this is 'risky' in that you now have all the keys published to the set of trusted technicians, but would speed up the process.
    Or failing that, the trusted technicians need a txt file or paper list of all the keys ...  [but will soon get fed up of entering 48 digit keys manually ...]

  • hitech9x's avatar
    hitech9x
    Brass Contributor
    Jul 21, 2024

    stephc_msftYou can grab the bitlocker keys using graph api 

    Could use list to loop through and grab all of the deviceIds, put them in a table, then loop through with a Get and save all of the keys by deviceId.  From there you could export it to a csv for easier parsing from the tool.  From a security standpoint, at the end of this being useful, there's a beta command to rotate keys on managed devices. I'm thinking bulk key rotation.

  • dominikhuettling's avatar
    dominikhuettling
    Copper Contributor
    Jul 21, 2024

    Hi all, I´ve attached the ISO to a VM in Nutanix, and get the following error message, any ideas?



    EDIT: Got it fixed by injecting the Nutanix VirtIO Drivers.

     

  • Intune_Support_Team's avatar
    Intune_Support_Team
    Icon for Microsoft rankMicrosoft
    Jul 22, 2024

    Thank you again for all your feedback. Still catching up on some of the comments, so appreciate the community help here! We've released an updated recovery tool with two repair options (see above) and the blog has been significantly adjusted. Thank you!

  • SergioA1260's avatar
    SergioA1260
    Copper Contributor
    Jul 22, 2024

    For those who had problem with the script try to move the folder containing the script on the root of the c drive

  • thegreymouser's avatar
    thegreymouser
    Copper Contributor
    Jul 22, 2024

    BruceK825 -- re: trying to get the status from the manage-bde.exe -status command, did you try using the -protectionaserrorlevel switch?  This will send the status to the ERRORLEVEL variable (0=encrypted)

     

    manage-bde status | Microsoft Learn 

     

    This is from the script that I wrote for our techs: