Kelsey Hightower hosted a Twitter Space on Sigstore: a new standard for signing and verifying software on March 22, 2022.
In total, 846 listeners tuned in! A few of us from Chainguard participated in the discussion, and we were joined by Sigstore co-founders and other community leaders. The full list of participants is below:
The conversation was very generative and offered a solid overview to help developers better understand the Sigstore project and how they can better protect their own software development life cycle. Here are four of my key takeaways from the discussion. You can listen to the full recording (with captions) on Twitter.
Signing and Verifying Software is Important
Kelsey Hightower framed the conversation with the importance of signing and verifying software, which is what Sigstore provides through its tooling. Beginning with the analogy that when you are walking down the street and see a piece of chewed gum, you don’t pick it up and chew on it; yet this is essentially what many are doing with software — people are often not checking where it came from or anything about it, and just running it on their machine, or leveraging it as a dependency for the software they are writing. As the industry has matured significantly, Hightower explains that we are now at “that point where this is no longer acceptable.”
Bob Callaway jumped in to share some of the more technical details of Sigstore, explaining how it provides a signature check and verifies that software has not been tampered with. Based on an Open ID Connect email address, the Fulcio tool under Sigstore verifies the integrity of a given signature. This workflow is similar to asking a third party to verify your identity through proof of possession. Callaway added that you can think of this process as Fulcio taking a look at the signer’s driver’s license, putting that together with a cryptographic key-pair, and then verifying that that person is who they say they are.
Kim Lewandowski chimed in to say that Sigstore’s goal is to have everyone verifying their software. There is a joke in the developer community that people are copying and pasting code, using existing code or code they may find on Stack Overflow. She asked, “Could you imagine a world where we can copy and paste code and verify it, to know it’s coming from where we think it’s coming from?” Even if people were taking a lazy approach of copying and pasting code, it would not be hazardous for projects if that code was signed and verified. Ideally, no one would be running or depending on code that they could not trust. This would make the entire software ecosystem more secure.
Discussing what exactly it is that developers sign, Seth Vargo explained that while someone “theoretically could sign individual files, that seems a little bit excessive. Realistically, you’re signing some compiled artifact, or deployable thing that could be a container, it could be a tarball. But think of it as this logical wrapper around the files and the runtime needed to execute the code that you authored.” Hightower compared this to signing a finished cake rather than the ingredients that are used to make up the cake.
The good news is that developers need not sign each software artifact manually, as Tracy Miranda explained. Sigstore can be used within a CI/CD pipeline to do automated signing and verifying, further reducing friction for developers.
Ultimately, signing and verifying software artifacts enables the wider community to check and confirm that the software they are running or leveraging in their own software projects are written by the people who said they wrote it and that that software has not been tampered with since it was signed.
Attack Vectors in the Software Supply Chain are Everywhere
The importance of signatures within the software supply chain became more concrete as the conversation progressed. Kelsey Hightower discussed knowing the provenance of different components of software and compared it to an airbag recall on a car. He explained that many people who have had cars know that recalls are fairly common, and they may ask themselves how do the manufacturers know that they are driving a car that has a particular airbag in it? The answer is that “typically we tend to have a pretty good supply chain or way to track individual components.” So the car manufacturers can attest that the airbag they placed in the car was known to be good when they installed it, but three years later they are finding that there is a problem and now they are asking to replace it because people’s lives depend on it. Hightower brings home the analogy: “Just because your software was deemed secure today. That may not be true tomorrow. And when that ever happens, we need a way to go back and at least notify.”
While the security issue with the airbag was not an attack, it clarifies that vulnerabilities can occur in any component, and it is therefore important to continue to monitor each component that makes up the larger body. To learn about attacks specifically, Hightower asked Seth Vargo, “Where are the attack vectors in a supply chain?”
In response, Vargo stated, “Honestly, the short answer is everywhere.” He went on to describe how code gets from a local development laptop to production infrastructure, and how a potential vulnerability or misstep can occur at any point. He explained how crucial it is to have a manifest to quickly search and “retroactively know whether or not you’re affected by a particular vulnerability at each stage.” The supply chain graphic below, adopted from the SLSA framework, can help you visualize relevant threats.
Vargo explained how Sigstore can mitigate many of those concerns. Through providing attestations and verifications in various places of the supply chain, Sigstore enables developers to check and make sure that a particular human or a particular machine took an action based on the signing keys used to sign given artifacts. Hightower highlights that “real people write software and we need to find a way to trust who they are” and to be able to check the chain if something goes wrong.
Sigstore’s Business Proposition works for Open and Closed Software
Throughout the conversation, our host took some questions from the audience, and a few dealt with business applications and how Sigstore could support closed source or proprietary software. For businesses, Sigstore offers quite a few advantages, not least of all that it helps to make signing and verifying software more frictionless for developers.
Providing some context of the original thinking behind Sigstore, Bob Callaway shared how he and his co-founders Luke Hinds and Dan Lorenc worked to prioritize the developer experience. He explained that they did not need to develop a new cryptographic algorithm, but work to make the tool “clean and simple and easy, and ultimately remove all of the friction around the experience to encourage people to do the right thing.” Building Sigstore with a developer-first mindset supports wider business adoption due to a reduced burden on developers and others who work with software.
Kim Lewandowski added some specific proprietary use cases that she had learned about in talking with groups who have adopted Sigstore. She shared that for large organizations insider risk — where an employee or someone else with access to a code base seeks to jeopardize software or data — is a real risk so these companies work to understand where code is coming from, to trace it back to a developer, or to otherwise remediate issues that come up. For these organizations, running their own instance of Sigstore architecture as part of their infrastructure with their own root authority to deliver the certificates internally is very compelling and something they are opting to do.
Continuing the discussion on proprietary software, Seth Vargo pointed out that even for those who are building a completely proprietary offering there is a zero percent chance that some piece of open source software is not a dependency of some kind — whether that is a library or the language runtime itself. “So,” Vargo explained, “even if you choose to completely ignore Sigstore for your own internal use, there’s still value in validating those external signatures.” Agreeing with Lewandowski, Vargo said that there is no reason why Sigstore could not be extended for proprietary use cases.
Giving the example of one company she has been talking to, Tracy Miranda recounted a use case that included the development of both open source and closed source software in-house. The organization in question has been using an internal signing system which they opted to move away from because of the problems that arise with key management. For this company, having the same signing flow through leveraging Sigstore was very appealing, and they ultimately chose to use the public transparency log because their customers could then also verify those signatures. Miranda summed up that Sigstore’s developer experience, as well as a consistent flow and signing mechanism across all software artifacts, can be appealing to companies whether they are in the open source or proprietary software landscape.
Community is the Heart of Sigstore
Sigstore’s community has really helped the project grow momentum and has enabled wide adoption. Bob Callaway called out that “Hundreds upon hundreds of people have joined, put up new requests, and got engaged in Slack channels and email threads... And it’s been really, really exciting to see over the past couple of years.” Kim Lewandowski added that she hoped that companies who use Sigstore contribute back to the open source project as well because “that’s how we all flourish and do great things.”
Matt Moore discussed some of the ways that the Sigstore community has organized itself to drive this success. He discussed how asynchronous communication has been emphasized, including recording calls, working in an open system with everything being discoverable on GitHub, using Slack and mailing lists. He also talked about the standards and documentation for how people can ask questions and get involved. Moore also explained how collaborators and contributors to Sigstore have taken off their vendor or company hats to prioritize the upstream team when working together on the project. He said that thinking this way helps to avoid upstream friction, and also helps people think of other upstream collaborators as effective coworkers. This inclusive environment also formed a lot of friendships over the years among contributors.
Wrapping up the discussion on community, Luke Hinds shared that when the community comes together to collaborate on an issue, the project can spread like wildfire, and this has been the case with Sigstore. He likened Sigstore to “a painkiller rather than a supplement;” something that people need for a specific problem. Hinds has found that many different open source communities — from Kubernetes to Python to Ruby — have started to converge and leverage Sigstore early on for software signing and provenance. He emphasized some of Moore’s points on how important it is to be “open and transparent in every facet of how you operate.”
When a great community converges around a technology, Hinds explained, then it is possible to achieve a lot. If you center the community and keep what is best for the community at heart, Hinds said, “then you can’t go too wrong.”
Quotes have been edited for clarity. You can listen to the entire Sigstore conversation on Twitter. Banner photo by Edgar Moran, cake ingredients photo by Calum Lewis, signature photo by DocuSign, laptop photo by Ales Nesetril, via Unsplash.
If you would like to join the Sigstore community, you can learn more from the Sigstore website, and join Slack and other communication channels through the Sigstore community repository on GitHub.
