A lot of ink has been spilled on the immutable and sometimes-mutable nature of container images. In some ways, it's really the perfect topic for hot takes and sometimes-dubious best practices. I'd like to spill some more ink, and introduce a novel solution to the problem, taking advantage of Sigstore's Rekor transparency log.
To solve this, the container ecosystem has relied on content-addressed image references, which rely on a foundation of strong cryptographic principles to generate a unique identifier for the image based on the exact contents of the image. If one bit is changed anywhere in the image, its digest changes, representing a completely new image.
Image Tags
And this works, more or less, so long as you trust users not to update those tags to point to something else. Or accidentally make a typo. Or have their credentials stolen by an attacker. Oh and you also have to extend this trust to all the systems and human operators running the registry service.
Here at Chainguard, when it comes to ensuring the integrity of our software supply chains, we're not in the business of having just a reasonable level of confidence. Hope is famously not a strategy. There has to be a better way.
Immutable Image Tags
Another approach to this problem has been immutable tags.
Some registry implementations, such as Amazon's Elastic Container Registry, allow users to configure their image repositories to enable immutable tags. In this scenario, only repo administrators can enable immutable tags, and only administrators can make tags re-mutable. While the repo has immutable tags enabled, users can only push new tags, and are blocked from updating existing tags.
And this works, more or less, so long as you trust your administrators not to forget to enable immutable tags on a new repo. Or make the repo mutable during maintenance and forget to flip it back. Or have their credentials stolen by an attacker. Oh and you also have to extend this trust to all the systems and human operators running the registry service.
Here at Chainguard, when it comes to ensuring the integrity of our software supply chains, we're not in the business of trusting puny human administrators not to mess up. To err is human. There has to be a better way.
Transparently Immutable Tags
Rather than trusting human users administrators, we can rely on transparency. Specifically, we can rely on Sigstore's Rekor transparency log, which holds an append-only log of signed attestations about code artifacts. Every entry in Rekor is publicly readable, and holds a cryptographic reference to all previous entries, so any modification to any previous entry would be easily and immediately tamper-evident. Because it's public, anybody can run a monitor to periodically check for bad entries, and alert on any inconsistencies.
Traditionally, Rekor has been used to store signatures and information about how artifacts were built, by whom, and what humans and systems attest to that. But Rekor is very flexible, and we can also use it to store attestations about what we've seen in the past, like what image tags point to what image contents.
Enter ✨tlogistry✨ – a public registry image proxy that collects tag history and stores it transparently in Rekor.
The next time you pull that image by tag, tlogistry will find that entry in Rekor, ensure that entry was previously signed by itself, and block any requests for that image by tag that don't match the previous digest.
Demo
Pull it through tlogistry:
(Note that the digest of the image we pulled matches the digest of the image we pushed.)
Build and push again, ⚠️ updating the tag️ ⚠️:
Pull it again through tlogistry, which fails because the tag changed! 💥
Because the source of truth is a publicly auditable, append-only transparency log, any user can query Rekor to see what tags tlogistry has already seen, and what digests they point to. Because tlogistry can only block updated tags, there's no trusting human administrator with access to enable/disable/re-enable it.
When you pull through tlogistry, image contents are still served directly from the original public registry, and container image data is never stored by tlogistry. Requests for image manifests by digest are unaffected. Only public images are supported.
When tlogistry serves an image manifest by tag, the HTTP response will include additional headers describing where exactly that information can be found in Rekor:
tlogistry is an experiment to demonstrate that transparency logs like Rekor can be used for more than just signing and attesting artifacts at build-time, and can be used to secure more stages of the software supply chain, even if the software's originators don't use Sigstore or Rekor. If this has inspired you to take advantage of Rekor in new and interesting ways, we'd love to hear about them!
The source for tlogistry is available on GitHub, and includes a Terraform script to build and deploy your own instance to Google Cloud Run, in your own project.
