Can Databricks bridge the open format divide?

Welcome to Runtime! Today: Databricks CEO Ali Ghodsi promises to unify the open-format split, a former Microsoft employee reveals new details about its broken security culture, and the latest enterprise moves.

Was this email forwarded to you? Sign up here to get Runtime each week.)

Peace in our time

SAN FRANCISCO — During the back half of the two-week Data Nerds Unite festival in San Francisco, it's been hard to avoid comparing and contrasting Snowflake's Data Cloud Summit, which took place last week, and Databricks' Data + AI Summit, which is occupying the same space at the Moscone Center this week. But the two companies actually have far more in common at this point than they did a few years ago now that Snowflake has embraced open storage formats.

Still, it was Databricks' turn in the spotlight this week. In a press conference Wednesday, CEO Ali Ghodsi promised that Databricks will work to make the competing Delta Lake and Iceberg formats "100% interoperable" over the next few years after acquiring Tabular last week.

Databricks introduced the open-source Delta Lake format back in 2019, but Apache Iceberg has gained a lot of followers in recent years, including Snowflake, which has revamped its product strategy around that format.
Tabular was founded by the original creators of Iceberg, and it was working on a managed version of what it calls "the leading open table format" until its acquisition by the biggest force behind Delta Lake, which accounts for 92% of the activity on Databricks' platform, Ghodsi said.
The formats are not compatible, but "the people who have created this technology … they'll admit themselves that the differences are nonsensical," Ghodsi said.
"Our ulterior motive with [the Tabular acquisition] is to bring the formats closer so that we can get interoperability between these two formats, Apache Iceberg and Delta Lake." he said.

At the same time, this is "a very difficult technical question that nobody on the planet right now has the answer to," Ghodsi acknowledged Wednesday. Databricks has a product called UniForm that allows customers to read tables across the different formats, but there's clearly demand for a more unified approach.

Tabular co-founder and CEO Ryan Blue, one of the original creators of Iceberg while working at Netflix, joined Ghodsi on stage Thursday morning for a brief discussion of the history of Iceberg and the need to unify the formats.
"Formats have always been a way for us to take on more responsibility as a platform and take responsibilities from people who worry about things," Blue said, but dueling formats make people worry about whether or not they bet on the right horse.
"We want to make sure that everything is compatible, that we're all running in the same direction with the single standard, if possible. Hopefully we can get there," Blue said.

But while Ghodsi and Blue figure out how to move the format question forward, Databricks continues to improve its catalog product — which was also a big part of Snowflake's announcements last week — and introduced a new business intelligence product.

Data catalogs sit a level above raw data stored in formats like Delta Lake or Iceberg and help data engineers ensure they can modify data in those tables concurrently.
Databricks announced Wednesday that it will open source its Unity catalog, which it introduced in 2021.
It also took the wraps off a product called AI/BI, a business intelligence dashboard that will compete with the likes of Tableau and Google's Looker.
Databricks has long been known as the data scientist company, but products like AI/BI are designed to expand its presence across its enterprise customer base into departments run by people who aren't mathematical geniuses.

Databricks is now on a pace to record $2.4 billion in annualized revenue, it revealed Wednesday. That's just shy of the $2.6 billion in revenue that Snowflake recorded during its 2024 fiscal year, but Databricks is growing faster.

"As long as there's vendors, they're going to compete, and there's going to be a fight between the different vendors. That's going to happen until the sun burns up," Ghodsi said Wednesday.
But Databricks engineers will have to work closely with Snowflake engineers and the rest of the Apache Iceberg community if it truly wants to unify the storage formats while keeping them open, and despite the obvious rancor between the two companies Ghodsi promised that will happen.

Active disaster

Concerns about the security of Microsoft's cloud and enterprise software products has been a running theme in cybersecurity circles for years, long before the company promised to rededicate itself to security following an embarrassing series of hacks against the U.S. government. But a new report from ProPublica reveals that Microsoft ignored internal warnings about the security of a key product because it worried that the fix would prevent it from winning the JEDI cloud computing contract.

Former Microsoft security engineer Andrew Harris laid out the story of the SAML flaw, a weakness in Azure Active Directory that could allow anyone who gained control of an on-premises server to get access to that organization's cloud resources. Harris urged Microsoft's product team to fix the issue for years starting in 2016, but his concerns were dismissed in part because a temporary solution to the problem "could also undermine the company’s chances of getting one of the largest government computing contracts in U.S. history."

Outside security researchers later discovered and published details about the flaw, and the SolarWinds hackers eventually used it to steal data from Microsoft 365 accounts, at which point Microsoft implemented the fix Harris had recommended. The report was published right as Microsoft President Brad Smith was set to testify to Congress about Microsoft's renewed community to security, and serves as just another reminder of how far Microsoft has to go to regain trust in its software.

Enterprise moves

Manish Gupta is the new chief marketing officer at LaunchDarkly, following two years in the same role at Sonar.

Lauren Nagel is the new vice president of product at Mezmo, joining the data telemetry companies after serving in a similar role at StackHawk.

Oliver McVeigh is the new chief delivery officer at Stibo Systems, and Jesper Palm Mortensen is the new chief operating officer at parent company Stibo Software Group.

The Runtime roundup

VMware revenue fell by $600 million year-over-year during its first quarter under Broadcom's direction, but costs went down by $700 million, which in Broadcom's worldview likely counts as a win.

Lynn Conway, a groundbreaking chip researcher and transgender trailblazer during a very conservative era of the tech industry, died last Sunday at the age of 86.

Thanks for reading — see you Tuesday!