This morning’s news of combined Microsoft IT outages alongside what was initially thought to be a system update from CrowdStrike are causing widespread disruption globally to organisations.
This does not appear to be a security incident or cyber attack – instead the CrowdStrike issue appears to have been caused by a faulty channel file related to the Falcon Sensor with symptoms experienced including Blue Screen of Death. CrowdStrike have issue an advisory but you have to be a customer to view the article. The result of the blue screen prevents machines being able to reboot.
The Falcon Sensor is designed to prevent attacks on systems whilst whilst recording activity of malicious behaviour for future review.
Director of Threat Hunting Operations at CrowdStrike announced a workaround on his X Channel here
His instructions for a workaround include the following actions.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching “C-00000291*.sys”
4. Boot normally.
IT Departments around the world will be experiencing a difficult period today and for the days to come whilst they resolve this. The fact this will involve a lot of manual tasks across 1000’s of systems in a process that can’t be automated engineers are going to have to visit sites to resolve the issue.
The CrowdStrike application is available in the Microsoft Azure Marketplace and it is possible that a lot of the cloud based Azure environments hosting business critical applications are possibly being impacted by this CrowdStrike problem taking down critical infrastructure for Healthcare, Airlines and Banking amongst others.
Organisations will no doubt be reviewing their Business Continuity plans off the back of this.
At the time of writing CrowdStrike have issued a statement stating the cause has been identified and a fix has been deployed – this may of course be too late for computers already impacted and needing manual intervention to resolve the Blue Screen of Death (BSOD).
Microsoft appear to be stating they are working on a fix with a third party vendor on the issue.
UPDATE:
CrowdStrike have issued the fix and isolated the problem, the issue will be the machines that have already had a Blue Screen of Death issue as it may need manual intervention to resolve. It’s now a manual process of resolving which means engineers in from ot computers. It’s going to take a long time to fix this.