More on the ClownStrike, er, CrowdStrike bugaboo problem

[thewayne]

What it really did was demonstrate bad IT practices, or IT shops that put entirely too much faith in their vendors (I could name a couple....)

The best practice for deploying an update is to have a computer lab that is isolated from your user/production network. Push the patch there, see what happens. Have a mix of machines in that environment. And with the proliferation of using virtual machines, it's not hard to do. You can have a mix of servers and workstations and different operating systems. THEN if everything works well there, push it out to a SUBSET of your production network.

Clearly that isn't what a lot of people did. They trust CrowStrike and just blasted it out. After all, it wasn't a code update, it was just like a virus update. What could possibly go wrong?

The problem was the update crashed the CrowdStrike driver, resulting in a blue screen of death upon reboot. And if the machine had an encrypted hard drive, it required manual intervention by IT boffins. All you had to do was delete one little bitty file, but you might not have had access to said little bitty file, particularly if said machine was encrypted.

Everything at the university yesterday seemed fine when I got in to work, no emails from main campus about subsystems being down, so that was nice. And it only affected Microsoft machines. Linux and Mac were safe.

To compound matters, Microsoft had some problems with their Azure cloud service, unrelated to the ClownStrike problem.

https://krebsonsecurity.com/2024/07/global-microsoft-meltdown-tied-to-bad-crowstrike-update/

Comments

[bibliofile]

And noted in a link here, but which I've seen noted around social media: Crowdstrike CEO George Kurtz was also CTO of McAfee during the big 5958 DAT problem caused an enormous outage back in 2010.

So, someone made a decision to push the update without enough testing? Quelle fucking surprise.

[thewayne]

Oh, how interesting!  Personally, I never cared much for McAfee or Symantec.  For PCs I use Zone Alarm for firewall and antivirus/malware.

[darkoshi]

https://www.crn.com/news/security/2024/crowdstrike-has-been-doing-updates-this-way-for-many-years-what-went-wrong

ThreatLocker CEO Danny Jenkins said it appears that the CrowdStrike update was not staggered because it was not a full software patch, which would have been released in stages.

Instead, Jenkins said, this was an update to CrowdStrike Falcon likely targeted at protecting customers against newly discovered cyberthreats, which is a frequent type of update for an endpoint security tool.

To keep customers protected, CrowdStrike “wants to push those threat updates instantly, to as many people as possible,” he said.
...
Hammond noted that as part of keeping up with hackers, many cybersecurity vendors have adopted similar practices around automated updating. Access to the Windows kernel—which has been implicated in the Microsoft outage—has also been considered crucial in order to provide strong security, he said.

The conglomeration of factors that made the outage possible is really “the nature of the beast” in terms of today’s cybersecurity practices, Hammond said.

Based on these pages (https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ and https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/), Crowdstrike pushed the bad update out at 12:09am Eastern U.S. time (0409 UTC), and pushed the fixed version out already at 1:27am (0527 UTC), and the problem only affected computers that were online during that 1.5 hour period.

I did not encounter a problem on Friday though several of my colleagues (including ones in India) did. I may have just missed it as per my notes, I finished my work up at 12:10am and must have put my laptop to sleep right around then.

[thewayne]

Interesting!  Amazing how timing can be the deciding factor.  I prevented our network at the police department from getting slammed by I Love You back in the '90s by reading a Slashdot post on it at 7am as part of my 'get to work' routine, and we pulled our firewall connection to the City to isolate ourselves and had pretty much zero infection from it as a result.  Sometimes you get lucky, sometimes you get slammed.

[disneydream06]

So the world was basically brought down by company and one big mistake. :o :o :o
Hugs, Jon

[thewayne]

According to one timeline in a comment on the previous post regarding the incident, a fixed update was released an hour after the buggy release.  If your computer was on when the bugged release was distributed, it was bombed and would have to be fixed.  If your PC was off until after the fixed release got out, when you turned it on it would be okay.  The problem is, servers are usually on and connected 24/7/365, so unless someone had a maintenance window and had their network down, or quarantined updates before releasing them, they were hosed.  Me, my work PC is normally on 24/7: I only turn it off when I'm going to be out of town or when I'm told to by IT.  My home PC is off when I'm not using it, though it turns itself on for backups at 7am.

[disneydream06]

I usually just put mine in sleep mode unless it's calling for an update and then I tell it to update and shut down. :o

[thewayne]

I hibernate my PC, which is a deeper sleep.

[disneydream06]

I wish I could hibernate. LOL!!!!!!!!!!!!!!!!!!!!

[arlie]

Does CrowdStrike allow its customers to choose the timing of updates, or accept them on some but not all of their systems? Does it charge substantial extra money (on the scale of a small customer) for licenses for those extra test systems? Can you be sure the test systems will hit the same conditions as the live system? (I *presume* CloudStrike had test systems of their own, which did not show the problem before the update was deployed to customers.)

Also, of course, by the time some youngster starts an IT career, they've been personally subject to forced updates for at least a decade, on their cell phone and personal computer. Unlike old fogies like me, they don't regard those updates as an imposed game of Russian Roulette - they regard them as not just normal, but in some sense the only way to distribute software, or at least the only modern way.

Remember, computers are not supposed to be reliable. Of course they crash regularly, etc. etc. ad nauseam. I'm glad to be out of that industry.

Editted to add: I see above that the mechanism used for the update gave customers no control. Surprise!

[thewayne]

From the beginning I've found it interesting that CrowdStrike's own testing didn't see the crash.  To me, this says they either rushed this out the door, or there's a bad flaw in their testing process.  Me, none of my systems update automatically: I install them and reboot them when I want to do it!  And like you, I'm glad to be out of that industry, it's a lot more pleasant spending my remaining years as a librarian.  And normally licensing is on production systems, not test or development systems.  But it varies.

[silveradept]

Great Godfrey. Even for the short amount of time that the borked updates were causing problems, that's an awful lot of people who are going to be very unhappy at whatever process failed and allowed the bad thing to go through. And whatever other bad processes then accepted a bad update.