Whelp, someone is probably getting fired over that one.
If you haven't checked the news today, basically everything is broken. Everywhere. For everyone. Crowdstrike pushed an agent sensor update and proceeded to BSOD every device with the Crowdstrike agent on it.
Oopsies. Hopefully CS gets their stack fixed in a rapid fashion.
IF YOU ARE AFFECTED, PLEASE USE THE FOLLOWING!
Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
Windows hosts which are bought online after 0527 UTC will also not be impacted
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.
Workaround Steps for public cloud or similar environment including virtual:
Option 1:
Detach the operating system disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
Attach/mount the volume to to a new virtual server
Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server
Option 2:
Roll back to a snapshot before 0409 UTC.
Workaround Steps for Azure via serial
Login to Azure console --> Go to Virtual Machines --> Select the VM
Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect" --> Click : "Serial Console"
Step 3 : Once SAC has loaded, type in 'cmd' and press enter.
type in 'cmd' command
type in : ch -si 1
Press any key (space bar). Enter Administrator credentials
Type the following:
bcdedit /set {current} safeboot minimal
bcdedit /set {current} safeboot network
Restart VM
Optional: How to confirm the boot state? Run command:
wmic COMPUTERSYSTEM GET BootupState