# Crowdstrike failure - How? Why? **URL:** \ **Category:** Factual Questions\ **Created:** [July 20, 2024, 1:54pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492 "2024-07-20T13:54:58Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![markn\_1](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@markn\_1](https://boards.straightdope.com/u/markn_1)\ **Post date:** [July 21, 2024, 2:08am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/41 "2024-07-21T02:08:53Z")
I only watched a little of that video, but I think the video just shows a screenshot of the start of the hexdump; that is, it’s just showing that the first 400 bytes of the file are all zeros. Does he somewhere confirm that the whole file is actually all zeros? Or maybe the whole file is intentionally all zeros and the “logic error” is a failure to handle that case. Although admittedly it’s hard to see how a file containing all zeros would convey useful information to the code. ---
**Author:** ![SunUp](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@SunUp](https://boards.straightdope.com/u/SunUp)\ **Post date:** [July 21, 2024, 3:38am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/42 "2024-07-21T03:38:19Z")
> [@markn\_1](#): > > Does he somewhere confirm that the whole file is actually all zeros? Starting at about 2:50 in the video he describes the file and says “the .SYS file that acted as an update to CrowdStrike somehow got shipped as all nulls” and then says basically the same thing again twice in the next 15 seconds. So he’s pretty much on the record as claiming that every byte in the file is zero. It’s very odd and it would be nice to get some clarification from CrowdStrike. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [July 21, 2024, 8:11am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/43 "2024-07-21T08:11:42Z")
> [@HMS\_Irruncible](#): > > I don’t know if Crowdstrike is hard-configured such that it can’t be remotely enabled to allow customers control over their update schedule. If that’s how it is, we now see how abundantly stupid this decision is. Security posture is important but not at the expense of operating continuity. I’ve never used Crowdstrike, but yeah, from experience ‘realtime’ updates are a specific selling point for some corporate security software (the notion being that the earlier you get the update, the lower the risk of falling victim to an attack) - and it was built right into the client software, with no option for staging or managing it. ---
**Author:** ![Cervaise](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cervaise/32/16693_2.png) [@Cervaise](https://boards.straightdope.com/u/Cervaise)\ **Post date:** [July 21, 2024, 9:58am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/44 "2024-07-21T09:58:54Z")
From a serious IT standpoint, giving up control is undesirable and risky. However, from a _management_ standpoint, vendor-driven updates are a big plus, because you can eliminate the IT staff that would otherwise be in charge of staging, testing, and releasing them. And even better, you have an outside party to point the finger at when things break. I work at a fintech providing analytical and reporting services to major investment institutions. I’m in charge of our contractual-compliance posture, ensuring we can deliver everything we say we can, from business product to resiliency. On the client side, the vendor-management teams could not possibly care less about the operational details on whether our services actually work. They care only that they can freely blame us in the event something goes wrong. Their technical people do care about how our platform is designed, how our technology integrates with the systems on their side, but contract ownership is a third-party oversight matter. We play nice with their tech folks, but contractually, we are not obligated to. Their management doesn’t care. Their management only wants a club to beat us with if we fail. I mean, I do agree with the principle expressed above. But short-sighted, incompetent management with misplaced priorities and malformed incentives is a major obstacle to healthy technology oversight in the modern marketplace, and appears to have contributed, in a broad sense, to this debacle. ---
**Author:** ![chappachula](https://avatars.discourse-cdn.com/v4/letter/c/d2c977/32.png) [@chappachula](https://boards.straightdope.com/u/chappachula)\ **Post date:** [July 21, 2024, 12:03pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/45 "2024-07-21T12:03:32Z")
> [@Stranger\_On\_A\_Train](#): > > a disgruntled employee Was this bug really so simple? Basically, I’m asking if it really is possible for just one person to crash the entire internet? ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [July 21, 2024, 12:16pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/46 "2024-07-21T12:16:47Z")
Sure, it’s possible for a single thing to break a lot of stuff. The software wasn’t on every part of the Internet, but it was present on enough of the component parts to break anything that is composed of multiple parts, and that’s most things. As for one person doing it, that really comes down to process within the software house, and the number of checks and balances that are in place, but even with all the right procedures in place, accidents can still happen because nothing in this universe can be perfect. Even supposing the update was fully tested and signed off and approved to go live, there are still ways it could go wrong - some of them exotic, others as simple as the person ultimately responsible for pressing the button selecting the wrong thing. My money is on someone in middle management yelling ‘why isn’t this release live? I don’t want to hear excuses! Do it NOW!’ and someone in release administration either caving to that pressure, or maliciously complying, knowing it could go wrong. ---
**Author:** ![GuanoLad](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/guanolad/32/42_2.png) [@GuanoLad](https://boards.straightdope.com/u/GuanoLad)\ **Post date:** [July 21, 2024, 12:19pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/47 "2024-07-21T12:19:56Z")
> [@chappachula](#): > > Basically, I’m asking if it really is possible for just one person to crash the entire internet? [![](https://imgs.xkcd.com/comics/dependency.png) ](https://imgs.xkcd.com/comics/dependency.png) ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [July 21, 2024, 12:20pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/48 "2024-07-21T12:20:04Z")
> [@Cervaise](#): > > However, from a _management_ standpoint, vendor-driven updates are a big plus, because you can eliminate the IT staff that would otherwise be in charge of staging, testing, and releasing them. And even better, you have an outside party to point the finger at when things break. Except in a case like this, no matter how the lawsuits and arbitrations and so on go, it’s not possible that the party to blame will be able to even come close to paying reparations on all of the damages they caused. ---
**Author:** ![Cervaise](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/cervaise/32/16693_2.png) [@Cervaise](https://boards.straightdope.com/u/Cervaise)\ **Post date:** [July 21, 2024, 1:02pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/49 "2024-07-21T13:02:54Z")
Totally agree. Now explain that to the people in management, who seem unaware that they’ll be in a queue of other claimants when a major failure condition occurs. ---
**Author:** ![TXRebel](https://avatars.discourse-cdn.com/v4/letter/t/6de8d8/32.png) [@TXRebel](https://boards.straightdope.com/u/TXRebel)\ **Post date:** [July 21, 2024, 1:07pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/50 "2024-07-21T13:07:51Z")
Wow! Thank you! That was so clear. I’ve been trying all weekend to understand this. ---
**Author:** ![griffin1977](https://avatars.discourse-cdn.com/v4/letter/g/977dab/32.png) [@griffin1977](https://boards.straightdope.com/u/griffin1977)\ **Post date:** [July 21, 2024, 1:23pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/51 "2024-07-21T13:23:23Z")
> [@griffin1977](#): > > There is a good breakdown here: > > [x.com](https://x.com/Perpetualmaniac/status/1814376668095754753?t=iCUQfCUwu9jwf2LRQnhFvw&s=19) So this in fact not a good breakdown, a clue would have been if I read to the end where he goes on a racist anti-DEI and anti-Rust\* rants. A non-racist and actually correct breakdown is here: > Its not missing a null check there is a null check in the disassembly. Its actually a table of pointers with some uninitialized entries. So the 156 byte value could be any value that happens to be in memory. `*` - I seriously don’t get this idea that the Rust programming language is somehow “woke”. Wtf is actually wrong with these idiots? I mean clearly a lot, and this is not the worst of them, but still… ---
**Author:** ![wolfpup](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/wolfpup/32/10618_2.png) [@wolfpup](https://boards.straightdope.com/u/wolfpup)\ **Post date:** [July 21, 2024, 6:13pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/52 "2024-07-21T18:13:35Z")
> [@griffin1977](#): > > Pointers are also the reason they are considered unsafe languages. As there is often no way to ensure that bit of memory being pointed to is actually a valid bit of memory containing what you expect Maybe a bit of a nitpick, but I don’t think that’s a valid criticism either of C++ or of what probably happened here. Every computer ever made uses pointers at the machine language level, whether it’s indirect addressing or something like the value in a register modifying the target of a memory reference instruction. And really, every high-level language uses pointers in some sense, even if they’re not called that. For instance, even in an ancient language like FORTRAN, array indices are essentially pointers, and if an array index goes wonky due to a programming error, the program will try to read or write in an inappropriate location, possibly outside of its own allocated memory. Of course it will fail due to memory protection, and will be stopped by the OS on an invalid memory access trap. And therein lies the real unsafe aspect of the Crowdstrike code: the fact that at least part of it runs in kernel mode, or at least is self-evidently able to affect the behaviour of kernel mode components. This kind of code is effectively part of the OS and can do whatever it wants, and thus has to be thoroughly tested. AIUI either this code wasn’t, or something in the deployment infrastructure wasn’t. And trust me, the deployment infrastructure can sometimes be a problem, especially if it’s old and creaky. ---
**Author:** ![leahcim](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@leahcim](https://boards.straightdope.com/u/leahcim)\ **Post date:** [July 21, 2024, 6:17pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/53 "2024-07-21T18:17:10Z")
> [@griffin1977](#): > > A non-racist and actually correct breakdown is here: Unrolled the thread at [Thread By @taviso - This strange tweet got \>25k retweets T..](https://unrollnow.com/status/1814762302337654829) for non-twitter users. > [@griffin1977](#): > > I seriously don’t get this idea that the Rust programming language is somehow “woke”. In programmer culture being anti-woke overlaps a lot with “I use a hard language so I’m better than everyone” coder-bro culture. Any language that makes programming easier is just letting less-hard-core coders in. ---
**Author:** ![RitterSport](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/rittersport/32/6326_2.png) [@RitterSport](https://boards.straightdope.com/u/RitterSport)\ **Post date:** [July 21, 2024, 7:16pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/54 "2024-07-21T19:16:15Z")
> [@smithsb](#): > > Explain as you would a child! Bonus point for the movie reference. Galaxy Quest. ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [July 21, 2024, 7:27pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/55 "2024-07-21T19:27:48Z")
> [@wolfpup](#): > > Maybe a bit of a nitpick, but I don’t think that’s a valid criticism either of C++ or of what probably happened here. Every computer ever made uses pointers at the machine language level, whether it’s indirect addressing or something like the value in a register modifying the target of a memory reference instruction. And really, every high-level language uses pointers in some sense, even if they’re not called that. Although all compiled machine code does essentially use pointers, it is typically managed either by the complier or by a dedicated memory management routine to assure that the programmer isn’t tripping over their own genitalia by either not properly initializing a pointer, dereferencing the wrong memory address, or allowing a memory buffer overrun (unintentionally using up all available allocated memory or using memory dedicated to system processes). Of languages in common use today, only C/C++ and some of the derivatives permit dynamic allocation of pointers and use of pointer arithmetic without some strong typing or trapping, and most robust languages have some kind of garbage collection so the programmer doesn’t incidentally induce a buffer overrun in some untested edge case. People here and elsewhere have really critiqued Fortran, for instance, for being obsolete even for computationally intensive programming, but you essentially can’t getting a buffer overrun in Fortran without really trying, and most Fortran programmers never have any need for pointer functionality even though it was added in at Fortran 95. Why use pointers at all? In theory, they can allow for much faster and efficient use of dynamic memory. However, while that may have been important in the ‘Eighties and early ‘Nineties, today dynamic memory is rarely the bottleneck, and the memory management of most modern languages adds almost negligible overhead to memory operations. The biggest practical reason is if you are running an application that runs in (near) real time and has to be absolutely deterministic; something like critical flight software or control of a nuclear reactor, for instance, where latencies may cause real physical problems. But these are very ad hoc systems, often using proprietary hardware that trades performance for high reliability and long uptimes, and as a consequence the software is extensively dynamically tested using a representative hardware testbed (“hardware in the loop” or HITL) to vet out both any logic problems in the algorithmic implementation and physical latencies in the hardware. There is rarely a reason that pointers have to be used in most commercial, non-safety critical applications, which is why most languages in use today have either completely eschewed pointers or allow them only within a protected and statically defined environment to prevent accidental overflows or other problems. > [@leahcim](#): > > > [@griffin1977](#): > > > > I seriously don’t get this idea that the Rust programming language is somehow “woke”. > > In programmer culture being anti-woke overlaps a lot with “I use a hard language so I’m better than everyone” coder-bro culture. Any language that makes programming easier is just letting less-hard-core coders in. Rust itself isn’t any more “woke” than any computer language but the Rust community is highly vocal about progressive politics, which has butt-hurt the significant portion of the programming industry that finds the inclusion of politics in programming objectionable even though most of them are happy to go on at great length about their typically sophomoric anarcho-capitalo-libertarian philosophies and weird obsessions with Ayn Rand and late stage Robert A. Heinlein. Rust itself purports to address a lot of the problems with performance and non-determinacy of automatic memory management at the cost of having to learn a new paradigm of programming, which if that what being “woke” is, I’m happy to wake up and stop worrying about error trapping every time a pointer is dereferenced. The “My language is harder and produces applications more likely to blow up if you don’t hold your mouth just so,” is deep within the C++ programming community, and they enjoy nothing more than sneering at anyone who uses a language with weak typing, imperative execution, intrinsic documentation and formatting, or anything else that makes a language more accessible to newbies or infrequent users. As I do most of my programming these days in Python, I get a lot of shit of this kind even though I’ve written many dozens of significant scientific and engineering applications with thousands of lines of highly robust Python code incorporating error trapping, high performance libraries, and which by the way can be read and understood as is with minimal documentation (just docstrings at this point). But then, Python doesn’t make a point of trying to intentionally create the most obfuscated code possible just as a point of obtuse pride. I’d personally be happy to never see a line of C++ (or Java) again, and if I needed to do any real bare metal or highly deterministic programming, I’d probably either go back to Fortran 90 or dive into Rust rather than wade around the the warm gasoline pool of C++. Stranger ---
**Author:** ![leahcim](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@leahcim](https://boards.straightdope.com/u/leahcim)\ **Post date:** [July 21, 2024, 11:20pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/56 "2024-07-21T23:20:12Z")
Discussion from a retired Microsoft engineer – [https://www.youtube.com/watch?v=wAzEJxOo1ts](https://www.youtube.com/watch?v=wAzEJxOo1ts) ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [July 21, 2024, 11:37pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/57 "2024-07-21T23:37:08Z")
[Moderating] While I’m sure that the wokeness or lack thereof of Rust is a fascinating topic, it’s a topic for a different thread. Discussion of how C/C++ handles pointers is on topic for this thread, since it seems to be a key part of how this failure occurred. Discussion of how C differs from other programming languages in this regard is probably also relevant, because that’s likely part of why the programmers screwed up here, because they might not have been familiar with C’s habit of letting you shoot yourself in the face. But save the stuff about woke programming for another thread, in another forum (probably IMHO would be the right forum for it). ---
**Author:** ![markn\_1](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@markn\_1](https://boards.straightdope.com/u/markn_1)\ **Post date:** [July 21, 2024, 11:50pm UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/58 "2024-07-21T23:50:32Z")
The Microsoft engineer is using the dump posted on Twitter that shows a dereference of 0x9c, and he guesses that it’s a null pointer dereference, which is a reasonable guess. But in the link that griffin1977 posted above, Tavis Ormandy notes two reasons to doubt that conclusion. First, the faulting instruction is ``` mov r9d, [r8] ``` rather than ``` mov r9d, [r8+0x9c] ``` This is suggestive but not conclusive. But second, in another dump seen by someone else, the dereference address is 0xffff9c8e00000008a rather than 0x9c, which cannot be the result of a null pointer dereference. It seems that the code is pulling a pointer from a table and dereferencing it, but it somehow pulled an invalid pointer. It’s probably uninitialized memory, which would explain why there are different fault addresses in different dumps. ---
**Author:** ![leahcim](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@leahcim](https://boards.straightdope.com/u/leahcim)\ **Post date:** [July 22, 2024, 2:34am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/59 "2024-07-22T02:34:26Z")
I do feel like focusing on the invalid pointer dereference is really burying the lede. Like saying someone died of “lack of oxygen to the brain” without mentioning that they bled out after crashing their plane which was shot down by the Red Baron. It’s the proximate cause of failure, but the situation went wrong _long_ before then. When people think of a null pointer dereference, the situation they usually think of is “programmer gets handed a pointer that might be null, and doesn’t check for null before using it”. From what I’m reading, this was not the problem here at all. The program was handed pointers that were supposed to be guaranteed to be valid, and they were not (but they were not null so they were not stopped from being used by null checks). Presumably these invalid pointers originated in the sys file that was part of the update. I don’t think there is a really effective defence against that on the code side. ---
**Author:** ![Stranger\_On\_A\_Train](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@Stranger\_On\_A\_Train](https://boards.straightdope.com/u/Stranger_On_A_Train)\ **Post date:** [July 22, 2024, 3:48am UTC](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492/60 "2024-07-22T03:48:13Z")
> [@leahcim](#): > > Presumably these invalid pointers originated in the sys file that was part of the update. I don’t think there is a really effective defence against that on the code side. No, but that is why mission critical code and all config files should be rigorously dynamically tested before release, and the release should be verified by (at least) a matching checksum before it is distributed. Clearly, there was a breakdown in this process with this CrowdStrike Falcon update. Stranger [Previous page](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492.md?page=2) [Next page](https://boards.straightdope.com/t/crowdstrike-failure-how-why/1004492.md?page=4)