CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (2024)

CrowdStrike would be feeling "very embarrassed" after issuing its Root Cause Analysis (RCA) of the faulty software update that led to potentially the largest global IT outage in history, experts say.

It came down to a mistake first-year programming students are taught how to avoid.

On July 19, the fateful Blue Screen of Death (BSOD) Friday, about 8.5 million Windows systems around the world went into meltdown when an update for CrowdStrike's Falcon sensor product went very wrong.

The US cybersecurity company released a preliminary report days after the incident.

Now a more in-depth, 12-page analysis has confirmed the root of the cause — one single undetected sensor.

Falcon's privileged access

CrowdStrike offers ransomware, malware and internet security products almost exclusively to businesses and large organisations.

The widespread outage has been linked to its Falcon sensor software, which is installed to look for threats and help lock them down.

Sigi Goode, a professor of information systems at the Australian National University, said Falcon had very privileged access.

It sits at what is called the kernel level of Windows.

"It's sitting as close to the engine that powers the operating system as possible," Professor Goode said.

"Kernel mode is constantly watching what you're doing and listening to requests from the applications you're using, and servicing them in a way that appears seamless to you."

He described kernel mode as the traffic police that Falcon sits alongside, saying, "I don't like to look of that vehicle, we should take a look at it".

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (1)

The sensor 21 culprit

CrowdStrike is constantly updating Falcon.

Loading Twitter content

On July 19, the company sent out a Rapid Response Content update to certain Windows hosts.

In the RCA, CrowdStrike called it the "Channel 291 Incident", in which a new capability was introduced into Falcon's sensors.

Sensors are like "a pathway for evidence," that tell it what sort of suspicious activity to look for, Professor Goode said.

"Falcon is looking at a range of sensors — a range of indicators — to see if something is wrong," he said.

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (2)

When updates are sent, it changes the location or the number of sensors to check for a potential attack.

In this instance, Falcon expected the update to have 20 input fields, but it had 21 input fields.

This "count mismatch" is what caused the global crash, CrowdStrike said.

"The Content Interpreter expected only 20 values," the RCA report states.

"Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash."

Because Falcon is so tightly integrated into the core of Windows, when it crashed it bought down the entire system causing the BSOD.

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (3)

Professor Goode said some of the most common ways to compromise a system were to flood memory.

Essentially, you tell the computer to look for something "out of bounds".

"It was looking for something that wasn't there," he said.

"But Falcon had to look in that 21st location, because that's what it was told to do by the new template it was given."

How can this happen?

CrowdStrike has apologised for the failure which has led to its CEO, George Kurtz, being called to testify before the US Congress to explain what happened.

"We are using the lessons learned from this incident to better serve our customers," Mr Kurtz said in a statement this week.

"To this end, we have already taken decisive steps to help prevent this situation from repeating, and to help ensure that we — and you — become even more resilient."

CrowdStrike's quality assurance (QA) processes have come into question.

The company has said that its updates "go through an extensive QA process, which includes automated testing, manual testing, validation and rollout steps".

But Rapid Response Content, which was used in this instance, goes through a different process.

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (4)

In the report, CrowdStrike admits that "lack of a specific test for non-wildcard matching criteria in the 21st field" contributed to "the confluence of these issues that resulted in a system crash".

Toby Murray, associate professor at the University of Melbourne's School of Computing and Information Systems, said the "dodgy data file update" was "embarrassing".

He said even basic checks by a human developer would have found the problem.

"That is an incredibly basic and fundamental mismatch that was always going to lead to catastrophic problems, sooner or later," he told the ABC.

"The fact that the CrowdStrike developers were able to have this obvious inconsistency between the data file format and the software code means that the most basic forms of quality review and assurance were not being correctly carried out."

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (5)

Professor Goode said this kind of mistake shouldn't be happening.

He said the update should have been released through a staged deployment.

"When they wrote this report, they must have been feeling very embarrassed," he said.

"First-year programming students are taught about the 'stack', the series of instructions that need to be executed in a CPU (central processing unit)."

CrowdStrike announced it had engaged with two independent software security vendors to conduct further review of the Falcon sensor code for both security and quality assurance.

Calls for accountability

In the wake of the outage, regulators and businesses have been considering legal implications.

The incident sent airports into chaos, supermarket check-outs stopped working, and media outlets struggled to bring you the news.

In Australia alone, the impact on businesses has been estimated at more than $1 billion.

Australian Industry Group CEO Innes Willox told ABC's The Business he expected the damage bill from the glitch to run into the billions of dollars.

But he said it was still unclear whether affected businesses would be able to seek compensation from CrowdStrike for any losses incurred from the outages.

America's Delta Airlines last week said the outage had cost the company $US500 million ($760 million) and that it planned to take legal action to get compensation from the cybersecurity firm.

CrowdStrike has rejected the claim, saying in a letter from an external lawyer that it is "highly disappointed by Delta's suggestion that CrowdStrike acted inappropriately and strongly rejects any allegation that it was grossly negligent or committed misconduct".

Delta cancelled more than 6,000 flights over a six-day period, impacting more than 500,000 passengers.

It faces a US Transportation Department investigation into why it took so much longer for it to recover from the outage than other airlines.

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about (2024)

FAQs

CrowdStrike reveals root cause of global outage, and it's a mistake first-year programming students learn about? ›

It came down to a mistake first-year programming students are taught how to avoid. On July 19, the fateful Blue Screen of Death (BSOD) Friday, about 8.5 million Windows systems around the world went into meltdown when an update for CrowdStrike's Falcon sensor product went very wrong.

When was the CrowdStrike outage? ›

The CrowdStrike Outage. On July 19, 2024, CrowdStrike, a prominent cybersecurity firm, faced an unexpected and severe outage that left many of its customers scrambling for solutions.

What is a CrowdStrike windows sensor? ›

CrowdStrike installs a lightweight sensor on your machine that is less than 5MB and is completely invisible to the end user. Once CrowdStrike is installed, it actively scans for threats on your machine without having to manually run virus scans.

How big is the CrowdStrike agent? ›

The CrowdStrike Falcon® platform is built on a lightweight architecture focusing on the “power of one,” integrating multiple advanced endpoint protection features within a single lightweight agent — less than 20 MBs in size — to deliver unprecedented efficacy against a wide variety of threats.

Why is CrowdStrike down so much? ›

On July 19, U.S. investors awoke to reports of what some experts were calling the largest IT outage ever. While investors were sleeping, CrowdStrike released a defective update to its software that caused Microsoft-based IT systems to go down.

What is the CrowdStrike issue? ›

CrowdStrike's software doesn't just run on Microsoft Windows; it also runs on Apple's macOS and the Linux OS. But the July outage only affected Microsoft Windows. The root cause of the outage was a faulty sensor configuration update that specifically affected Windows systems.

What is CrowdStrike famous for? ›

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas. It provides endpoint security, threat intelligence, and cyberattack response services.

Will CrowdStrike recover? ›

The good news: Investors often have short memories

As concerning as this may be, the episode may not necessarily be crippling to CrowdStrike's business. The company can recover, and while it seems horrible right now and damaging to the brand, stocks can and often recover from bad press.

What makes CrowdStrike unique? ›

The advantage of CrowdStrike is its advanced threat detection capabilities, real-time response, and cloud-native architecture, which allows for quick deployment and scalability.

Does CrowdStrike block malware? ›

For example, CrowdStrike is AV Comparatives approved, with a 99.2 percent malware block rate, and zero business false positives. In addition, the Falcon platform meets the compliance standards of PCI DSS Requirement No.

Who is CrowdStrike biggest competitor? ›

Top Competitors and Alternatives of Crowdstrike

The top three of Crowdstrike's competitors in the Endpoint Protection category are McAfee ePO with 21.36%, SentinelOne with 9.57%, Duo Security with 7.72% market share.

Is CrowdStrike an Israeli company? ›

Beyond the business rivalry, CrowdStrike is part of an exit strategy for many Israeli cybersecurity startups. The American company, which has a $4 billion reserve and wants to expand its solution portfolio, has become a target for Israeli venture capital funds looking for a buyer for their offerings.

Who owns CrowdStrike? ›

The ownership structure of CrowdStrike Holdings (CRWD) stock is a mix of institutional, retail and individual investors. Approximately 49.53% of the company's stock is owned by Institutional Investors, 2.19% is owned by Insiders and 48.28% is owned by Public Companies and Individual Investors.

When did CrowdStrike come out? ›

Founding: 2011–2019

The company launched CrowdStrike Falcon, an antivirus package, as its first product in June 2013. In May 2014, CrowdStrike's reports helped the United States Department of Justice to charge five Chinese military hackers with economic cyber espionage against U.S. corporations.

Does the US government use CrowdStrike? ›

Crowdstrike is in wide use across federal agencies and it is a key vendor on the governmentwide Continuous Diagnostics and Mitigation cybersecurity support services contract.

How much is George Kurtz worth? ›

What is the long term outlook for CrowdStrike? ›

The financial impacts of this expanding portfolio are reflected in CrowdStrike's future outlook. The company expects total revenue for Fiscal Year 2025 to be between $3,976 million and $4,010 million, representing a year-over-year increase of almost 10% from $3,650 million.

References

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Fr. Dewey Fisher

Last Updated:

Views: 6069

Rating: 4.1 / 5 (62 voted)

Reviews: 85% of readers found this page helpful

Author information

Name: Fr. Dewey Fisher

Birthday: 1993-03-26

Address: 917 Hyun Views, Rogahnmouth, KY 91013-8827

Phone: +5938540192553

Job: Administration Developer

Hobby: Embroidery, Horseback riding, Juggling, Urban exploration, Skiing, Cycling, Handball

Introduction: My name is Fr. Dewey Fisher, I am a powerful, open, faithful, combative, spotless, faithful, fair person who loves writing and wants to share my knowledge and understanding with you.