One of the most common – and most dangerous – practices we see is running outdated container images in production. I frequently read Internet comments such as “I just decommissioned my last Alpine 3.5 container,” or “I just got around to updating my images from Debian 8 to Debian 11”. Alpine 3.5 was declared end-of-life, meaning no more security fixes, over four years ago. Similarly, Debian 8 was declared end-of-life nearly 2 years ago. Because popular base images can have tens or even hundreds of reported vulnerabilities, constantly upgrading container images is an important way to stop a molehill of security debt from becoming a mountain.
Best practices versus reality: how we have gotten to the current state of software distribution
When Solomon Hykes released Docker in 2013, it had a revolutionary effect on the way software, especially software appliances, was distributed. Unfortunately, while the impact of Docker cannot be understated, many software appliances deployed in production utilize outdated container images, frequently due to the overhead cost of monitoring images for updates and vulnerabilities. When this monitoring does not happen, it results in a lose-lose situation: the development and operational velocity enabled by software appliances is frequently “paid off” through the security debt that comes from the lack of proper security lifecycle management for appliances deployed in production.
It doesn’t have to be this way, however. By properly leveraging continuous deployment and canary testing with Kubernetes, it is possible to have frictionless development while enabling the operations teams to sleep easily at night. At Chainguard, we are building a platform using open source tools, such as apko, Tekton, and mink to enable this reality.
Patch velocity is essential for minimizing developmental security debt
We all know keeping our systems up to date is critical, which is why distributions provide packages enabling automatic updates, such as Alpine’s apk-cron or Debian’s unattended-upgrades, but containers aren’t managed in the same way distributions running on bare metal or in VMs are managed. Instead, containers are rebuilt and replaced. In an ideal world, containers are short-lived and are constantly being refreshed, but historically the tooling provided has not been suitable for such a workflow.
To solve the patch velocity problem for Alpine and other distributions in the APK-based distribution ecosystem, we created apko: a declarative OCI image builder that generates reproducible images. By combining apko with a CI system, such as Github Actions, we have the basic components of a secure software factory that constantly refreshes the images used in production, keeping them clean of reported vulnerabilities. The Distroless Github project demonstrates this workflow, providing a set of experimental base images which are refreshed nightly with the latest security updates from Alpine.
Also, as highlighted in our whitepaper, when developers ship only the necessary components of a distribution to support their application, there are fewer total packages in the container image, which reduces the patching burden. Tools like apko when combined with cloud-native package building tools such as Melange provide the capability to build images that are leaner than the ones provided by distributions themselves.
When developers use these approaches to distribute their own software appliance images, they provide their users with clean, up-to-date container images that minimize security debt, which solves the supply side of the problem with less developer burden being spent on maintaining images.
Deployment velocity is important for minimizing operational security debt
While it’s true that developers need to do more – simply shipping a new software appliance every time a new version of the underlying software is released is not enough – operations teams need to update the appliances they have deployed to production. Due to the traditional model of deploying services on bare metal servers and virtual machines, operations teams find themselves surprised at the notion that containers should have short lifetimes: when I worked as a cloud-native ecosystem consultant prior to joining Chainguard, this was a significant area of confusion as operations teams made the jump to using containers.
Like your groceries, containers are meant to be short-lived. The average container age running on your production cluster should be measured in days, not weeks, months or years, because the ideal DevOps deployment should feature continuous deployment and functional testing before being admitted into production. While software appliances have made operations much easier, they are not foolproof: if you set up an appliance, and never touch it again, you are setting yourself up for a security incident.
Operations teams can look into using tools like Tekton to continuously deploy new versions of their images into production. When combined with Knative’s revision-based traffic steering, which allows for A-B testing of a given service, an operations team can have a high deployment velocity that does not harm the uptime of their applications.
How can I fix my security debt nightmare?
In short, a security debt-free container lifecycle requires high patch and deployment velocity. The use of tools such as apko (image building), Melange (package building), Tekton (continuous deployment) and Kubernetes canary deployments can equip hard-working DevOps teams with workflows and processes that drive the necessary culture change to get to zero security debt. If you’re interested in improving your patching and deployment velocity, reach out to Chainguard today – we can help!
