More on the ClownStrike, er, CrowdStrike bugaboo problem

[thewayne]

What it really did was demonstrate bad IT practices, or IT shops that put entirely too much faith in their vendors (I could name a couple....)

The best practice for deploying an update is to have a computer lab that is isolated from your user/production network. Push the patch there, see what happens. Have a mix of machines in that environment. And with the proliferation of using virtual machines, it's not hard to do. You can have a mix of servers and workstations and different operating systems. THEN if everything works well there, push it out to a SUBSET of your production network.

Clearly that isn't what a lot of people did. They trust CrowStrike and just blasted it out. After all, it wasn't a code update, it was just like a virus update. What could possibly go wrong?

The problem was the update crashed the CrowdStrike driver, resulting in a blue screen of death upon reboot. And if the machine had an encrypted hard drive, it required manual intervention by IT boffins. All you had to do was delete one little bitty file, but you might not have had access to said little bitty file, particularly if said machine was encrypted.

Everything at the university yesterday seemed fine when I got in to work, no emails from main campus about subsystems being down, so that was nice. And it only affected Microsoft machines. Linux and Mac were safe.

To compound matters, Microsoft had some problems with their Azure cloud service, unrelated to the ClownStrike problem.

https://krebsonsecurity.com/2024/07/global-microsoft-meltdown-tied-to-bad-crowstrike-update/

Comments

[darkoshi]

https://www.crn.com/news/security/2024/crowdstrike-has-been-doing-updates-this-way-for-many-years-what-went-wrong

ThreatLocker CEO Danny Jenkins said it appears that the CrowdStrike update was not staggered because it was not a full software patch, which would have been released in stages.

Instead, Jenkins said, this was an update to CrowdStrike Falcon likely targeted at protecting customers against newly discovered cyberthreats, which is a frequent type of update for an endpoint security tool.

To keep customers protected, CrowdStrike “wants to push those threat updates instantly, to as many people as possible,” he said.
...
Hammond noted that as part of keeping up with hackers, many cybersecurity vendors have adopted similar practices around automated updating. Access to the Windows kernel—which has been implicated in the Microsoft outage—has also been considered crucial in order to provide strong security, he said.

The conglomeration of factors that made the outage possible is really “the nature of the beast” in terms of today’s cybersecurity practices, Hammond said.

Based on these pages (https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ and https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/), Crowdstrike pushed the bad update out at 12:09am Eastern U.S. time (0409 UTC), and pushed the fixed version out already at 1:27am (0527 UTC), and the problem only affected computers that were online during that 1.5 hour period.

I did not encounter a problem on Friday though several of my colleagues (including ones in India) did. I may have just missed it as per my notes, I finished my work up at 12:10am and must have put my laptop to sleep right around then.

[thewayne]

Interesting!  Amazing how timing can be the deciding factor.  I prevented our network at the police department from getting slammed by I Love You back in the '90s by reading a Slashdot post on it at 7am as part of my 'get to work' routine, and we pulled our firewall connection to the City to isolate ourselves and had pretty much zero infection from it as a result.  Sometimes you get lucky, sometimes you get slammed.