Breaking news - huge outage/interruption created by Crowdstrike problem (July 2024)

It’s not; the company sells mostly or entirely to enterprise clients.

My understanding is that this is exactly the case. Based on discussion upthread, it looks like, while they do apparently offer a consumer product, the vast majority of their business is done with companies, not individual/home consumers.

So I see. I’ve been out of the software game for a few years now, but when I was in it, I was a kernel hacker more concerned with performance issues than security stuff.

I always did think though that security programs using a ‘scan for known badness’ approach were fighting an ultimately unwinnable battle, and the only reasonable method is a whitelist.

But users will download attractive stuff… :slight_smile:

Anyway, getting a bit off topic for Breaking News… if anyone is interested in discussing security theory we should probably spin off a new thread.

Can be a little bit of both. The last place I worked was a small shop, and the problem with small shops is that there is always a prima donna.

In this case, our prima donna was the golden child of the manager. He put a change out without going through any testing, which all the other developers would have sent to our very good test group. Of course it broke everything.

In the post-mortem, manager kept asking why the problem wasn’t caught in testing. After the second or third time, I said point blank that the tester couldn’t test a change he didn’t know was going in. I don’t think the manager liked that.

As a programmer, I think the testers are the real heroes.

Not quiet a C64 but almost. They didn’t get affected because they still run Windows 3.1.

Cite

My brain is getting old. I just can’t process that a company named Crowdstrike is focused on protecting you from security threats, not creating them. Its name should have some mutation of “protect”, “secure”, “guard”, or at least “umbrella” in it.

Are we even sure this wasn’t intentional? I don’t want to go through convincing my brain that Crowdstrike isn’t a bunch nihilist eastern European hackers, only to find out in a week or two that it actually is.

If so, they’ve done an exceptionally good job of covering their tracks, given that, in the 13 years since they were founded, they’ve repeatedly uncovered and addressed cyberattacks by Russian, Chinese, and North Korean hackers against Western targets.

Well it’s not as crazy as it sounds. The purpose of Crowdstrike is to stop hackers from causing harm. China is certainly working on ways to collapse our infrastructure through hacking methods. Give someone $1M to change a line of code that magically doesn’t get tested…

Not that I disagree, but there’s a large difference between one compromised programmer at Crowdstrike, and this idea:

I agree. But I was born at a time when physical tickets got you on a plane and worldwide scheduled flights didn’t collapse over a software bug.

Yeah, it sounds like it should be a tool for conducting distributed denial of service attacks (where a whole bunch of different users, probably a botnet of home computers taken over by some other security hole, all try to access the same site at once to overwhelm its capacity).

I wonder how much testing for small changes they do as standard procedure. Reducing work force size to increase profits can lead to this kind of thing. In this case, even the simplest test would have caught it.
Not unprecedented, though. When I was at Bell Labs one guy screwing up a C switch statement and not testing the revised module because the change was so small took down a good chunk of the long distance system.

Great. Now they want us to make an appointment for onsite IT support but their appointment scheduler appears to be overloaded! First, there were no appointments available until tomorrow and trying to book one of them continually returned “please try again”; now there are no appointments available at any time!

And IT’s phone is still going straight to voicemail.

I found this buried in the comment section of Ars Technica — a Crowdstrike blog entry from 2021 where they urge their customers to accept and implement urgent updates without testing them.

On the one hand, it could be argued that their evident confidence in their product was warranted at the time, because it took three and a half years after this article was published before a malformed update blew up the world. But now, obviously, this is not a great look.

Microsoft says 8.5 million Windows computers were affected. Starting at $9/month/seat for a Crowdstrike license, that’s some pretty good revenue. Before this incident I couldn’t remember if the company was Cloudstrike or Crowdstrike, so maybe any publicity is good publicity?

Microsoft has also released a bootable USB recovery tool.

Microsoft estimates there are 500 million (based on their educated guess regarding where numbers were heading late last year) devices running Windows 11 and another 200 million which haven’t been upgraded from 10 so that’s approximately 1%.

2 things:

  1. Delta is still having problems digging out from under this 3 days out because their crew scheduling computers took a hit.
  2. I vaguely remember a computer crash many years ago that gave me a boot choice of the last previous update or something along those lines. Vague is the key word. Am i misremembering this entirely?

I don’t know if this was in jest, but the first iPad appeared on the market in early 2010, 14½ years ago.

Windows has a “system restore point” capability that can checkpoint the state of the system files, allowing a fallback if an update doesn’t work.

But those are optional, they require time to execute and allocated file system space to store, and I don’t think most endpoint security products do them (perhaps because malware might be able to silently use a restore to roll back anti-malware software).

Was able to secure at IT appointment for this morning and am using it now; putting in a half day.

For timekeeping purposes, the use of Non-Classified Time has been granted for however long computers were down, within reason. “Within reason” determined by one’s immediate supervisor(s).