CrowdStrike's deployment disaster is a wakeup call

Welcome to Runtime! Today: why security software customers should hold their vendors to a higher standard, the Wiz kids turn down Google, and the latest funding rounds in enterprise tech.

(Was this email forwarded to you? Sign up here to get Runtime each week.)

Kernel power = kernel responsibility

Unless they were caught in the throes of a ridiculous 2019 conspiracy theory, most people heading to the airport last Friday morning had probably never heard of CrowdStrike. That changed in a big way after CrowdStrike caused the largest IT outage ever, delaying flights and shutting down banks with a bad software deployment.

Several days later, CrowdStrike has yet to fully explain the circumstances that led up to the deployment of a configuration file with a bad memory address to all Microsoft Windows users running its Falcon threat-detection and response software. According to Microsoft, that update crashed 8.5 million PCs and servers, which is a tiny number of Windows machines in the wild but included business-critical systems used by high-profile enterprise customers such as airlines, banks, and retail stores.

• Like most enterprise security products, Falcon requires near-constant updates to keep up with the ever-growing number of security threats.
• Enterprise security products built for Windows have access to the kernel — the core of the operating system — which allows them to see into any software applications running atop the operating system and detect and respond to security issues more quickly than possible without that access.
• Most modern operating systems restrict access to the kernel, but Microsoft was more or less forced to allow third-party security companies access to that layer out of concerns in the mid 2000s that Microsoft would corner the market for enterprise security software if it was the only company that enjoyed that level of access.
• However, when an update goes bad at the kernel level, it crashes the entire computer, not just the application itself.

Something went very wrong inside CrowdStrike's internal software deployment process to allow such an error to wreak havoc around the world. Most software updates — let alone kernel-level updates — are thoroughly tested to detect basic errors such as a bad memory configuration before they are released.

• The Pragmatic Engineer has a good example of how this usually works: Software updates are generally pushed to a small subset of users in a defined geographical area, and then rolled out more widely as it becomes clear everything is working as designed.
• For reasons it has yet to explain, CrowdStrike appeared to deploy the bad configuration file globally with the push of a button and didn't roll it back for a little over an hour, at which point the damage was done.
• Some security experts speculated that CrowdStrike's policies for configuration files might have been looser than ones applied to actual feature updates in Falcon because of the speed at which security software needs to be updated.
• But it's now clear that vendors who have kernel access can do a lot of damage with a single relatively innocuous file.

The incident will likely prompt a debate over whether or not Microsoft and regulators should change their stance on kernel access for third-party developers, and there are good arguments both for and against such a change. Still, CrowdStrike owns this failure, and any Windows customer working with an enterprise security vendor should demand far more transparency into the testing and deployment of kernel-level updates.

• "The whole situation feels like handing the keys to the kingdom — basically the global economy, such is Microsoft Windows' reach —to a small group of private cybersecurity companies with no external governance or assurance," wrote security researcher Kevin Beaumont in a blog post calling for customers to demand more accountability for their suppliers.
• As Beaumont noted, companies in highly regulated industries — such as airlines and banks — are often required to install managed security systems like CrowdStrike's Falcon to demonstrate their commitment to security, and are at the mercy of those vendors and their quality-control systems..
• "... we don’t know yet fully know what happened or why, but one thing I do know is that any company that builds any kind of software should design, test, and deliver it with a priority on dramatically driving down the number of flaws—flaws which can be intentionally exploited by bad actors or flaws that can unintentionally take down critical services across the globe," wrote CISA Director Jen Easterly on LinkedIn.

Enterprise independents 2, Google Cloud M&A 0

Rather than become the largest acquisition in Google's history, Wiz turned down its $23 billion offer Monday and pledged to go it alone. According to Bloomberg, its decision was inspired in part by increased tech regulation, which would have extended the review period for any deal that big, and the Crowdstrike incident, which just opened up a huge opportunity in cloud security.

Wiz, which has raised $1.9 billion in funding, provides a similar platform-style service to Crowdstrike but is focused exclusively on applications running in the cloud. Security startup funding fell off a cliff in 2023 as global tech spending retreated, but Wiz is already doing $350 million in annual recurring revenue and expects to hit a billion in the near future, according to a memo sent to employees Monday evening viewed by CNBC.

For Google Cloud, it's the second major deal to fall through just this month after Hubspot walked away from acquisition talks in early July. However, its core business continues to grow at a healthy 29% clip, exceeding analyst expectations for the second quarter and topping $10 billion in revenue for the first time.

Enterprise funding

Cohere raised $500 million in Series D funding valuing the AI model developer at $5.5 billion, and it celebrated by laying off 20 people.

Linx Security emerged from stealth mode with $33 million in new funding to build out its identity-management security product.

Momento landed $15 million in Series A funding for two former AWS engineers building a new take on real-time data infrastructure.

Resquared raised a $5 million seed round to build a Salesforce for local businesses. 

The Runtime roundup

Meta introduced Llama 3.1, a new version of its core large-language model that it compared favorably to OpenAI's GPT-4 and Anthropic's Claude 3.5.

SAP's cloud revenue jumped 25% compared to last year, as it continues to move customers off older on-premises applications.

Workers at Microsoft's renowned AI lab in China, which it has grown reticent to discuss in recent years as anti-China sentiment has grown in U.S. politics, are being pressured to move outside the country to other offices in Australia, Canada, and the U.S., according to Rest of World.

Thanks for reading — see you Thursday!