Who will maintain the maintainers? Open-source groups push for a sustainable future

The open-source software movement unleashed a torrent of innovation and economic activity by creating free, reusable building blocks for the internet infrastructure that runs the world's economy. More than two decades later, the maintenance bill for those projects is coming due.

Maintaining an open-source project is an extremely important and underappreciated job, the kind of work people take for granted until something goes wrong. Countless open-source projects have been used to build software, some of which are maintained by foundations with nine-figure annual budgets and some of which are maintained by one or two people in their spare time.

But the ongoing maintenance of those projects is about to become even more important in response to worries about software supply chain security. As of next week, the U.S. government will require suppliers to the federal government to produce a software bill of materials (SBOM) outlining all the code used to build their software, and in many cases, that involves tracking down details on a lot of different open-source projects.

"This is perhaps the most valuable supply chain in the world that's ever been built," said Luis Villa, co-founder and general counsel of Tidelift, during Tidelift’s Upstream conference Wednesday. The problem is open-source developers and maintainers didn't necessarily set out to become "suppliers," which implies an ongoing commitment to those projects that they simply might not have the time or interest in doing, he said.

Tidelift is trying to find ways to compensate maintainers for these additional efforts, including everything from cash to community support and training. But the idea of paying open-source maintainers does not sit well with everyone, because it can distort the incentives for working on open-source projects and allows big software companies to avoid responsibility for their product decisions.

"Supply chain security is a problem that exists mostly because the consumers are doing a poor job of understanding what dependencies they have, and then responding to them and making the updates," said Brian Fox, co-founder and CTO at Sonatype and governing board member of the OpenSSF, in an interview. "It is mostly not that the maintainers are doing a poor job updating those things."

The art of maintenance

All software — open source or commercially produced — requires maintenance.

Running code in different ways over time can produce bugs that would have been nearly impossible to detect during the initial build, and intrepid cybercriminals around the world are constantly poking at software to find holes they can exploit before maintainers can release a patch.

The discovery of the SolarWinds hack in late 2020 — followed next year by the race to patch the Log4j vulnerability in late 2021 — set off alarm bells in Silicon Valley and inside governments around the world. Those incidents led to the development of the Biden administration's push for SBOM requirements and the introduction of the Cyber Resilience Act by the European Commission, both of which require companies to attest to the security of their software.

But according to a recent survey conducted by Tidelift, more than half of open-source maintainers were not aware that new security standards had been developed both by industry groups like the OpenSSF and NIST, the federal body overseeing the SBOM rollout. And among the maintainers who were aware of the requirements, nearly 40% said they had no intention of following through and 19% said they weren't sure what to do.

"There is a lot of underlying work here that a maintainer is signing up for," said Lauren Hanford, vice president of product at Tidelift, in a presentation Wednesday outlining the results of the survey. Of the maintainers who said they weren't planning to do the work to adhere to the new standards, 38% said they didn't have the time and 37% said they weren't getting paid, so why bother, according to Tidelift.

It's important to distinguish between the different types of open-source projects when trying to get a handle on the seriousness of the software supply chain problem, said Chad Whitacre, head of open source at Sentry and co-creator of FOSS Funders, an attempt to build a coalition of enterprise tech companies willing to donate to open-source maintainers.

There are lots of open-source projects that are designed for end users, like Audacity, or managed by well-funded industry foundations, like Linux. In those cases, the question of who maintains those projects and how they are compensated is fairly well understood, Whitacre said in an interview.

But in the middle are countless numbers of developer tools, programming languages, frameworks, and other pieces of the plumbing that individuals and companies have been using to build software for a very long time. Some of those enjoy substantial commercial support, but a lot of them don't, and as one of the more famous XKCD cartoons demonstrated, those projects play key roles in the modern software stack.

Whitacre's project was designed to funnel contributions from individuals and companies using open-source software to those maintainers. FOSS Funders is just getting off the ground, but part of its mission is to make it easier for companies that want to help bypass frequently overlooked obstacles inside their companies, like how to get accounts payable to cut checks to individual maintainers.

"This is in all of our collective self interest to make sure that these things are sustained," Whitacre said. "Yes, the Linux Foundation exists, but there is this role for the small to medium businesses that really kind of depend more on this (software) to band together to pool our resources to make sure that we're sharing best practices and really supporting folks."

According to Tidelift's survey, 47% of maintainers said they would follow the new security guidelines if they were compensated for the effort to update their projects.

Bounty hunters

The combination of the new security laws and the push to directly compensate open-source maintainers is uncharted territory for this movement, and there are lots of questions about how effective both efforts will actually be when it comes to improving security.

The debate over open-source compensation is not a new one. Critics worry that it could distort the incentives for working on an open-source project by incentivizing work that isn't necessarily in line with the project's goals.

"The last thing you need is a bunch of mercenaries driving around trying to collect bounties for adding features to a project that they don't know anything about," Fox said. He argued that the focus on maintainer compensation actually distracts from the real supply chain security problem, which is that most maintainers already release timely updates that are ignored by end users.

"At best you're solving 4% of the problems today" by compensating maintainers to work on updating projects, Fox said. "Four percent is (end users) using vulnerable components because they don't have a better choice. 96% is literally they're ignoring the fixed thing that is sitting right next to the broken thing. We have to fix that problem."

When the next zero-day vulnerability hits, SBOM requirements should give those companies less of an excuse for being unable to quickly determine which software components they're using and where they are running. But Fox, and many others in the open-source community, are very concerned about maintainers facing potential liability for having their projects included in software they didn't assemble under Europe's Cyber Resilience Act, which could change the stakes of the compensation debate forever.

"That's really the problem: (In) open source, you can't know how your stuff is going to be used. So how could you possibly be held accountable for ensuring that it's safe in all those (future) uses? That's impossible," Fox said.

"If jurisdictions try to go down that path, the unintended consequences will be many people will just opt out," he said. "Because who's going to take untold liability for their entire livelihood for a project on the side that they get nothing but satisfaction out of?"

If governments do go down that path, Whitacre suggested they follow the lead of Germany's Sovereign Tech Fund, which launched earlier this year and sets aside tax revenue to "sustainably strengthen the open-source ecosystem."

"The effort to improve cybersecurity cannot just stop at defining and publishing new standards," Tidelift wrote in its report. "This data revalidates the need for investment, especially from organizations that rely on open-source components, in the open-source maintainer community," the report said, and those organizations certainly include governments.

"It's commercial suicide to try to build competitive software without using any open source in 2023," Fox said. The tech industry has thrown a ton of effort into demonstrating how committed it is to environmental sustainability, and at some point, it will need to make a similar commitment to software sustainability.