Secure Your Software Factory with melange and apko

Secure Your Software Factory with melange and apko

The GitHub repo associated with this blog post can be found here.

What is a “Secure Software Factory”?

For simplicity, let's imagine a typical brick-and-mortar factory that produces toys. A software factory is sort of like that, but at the end of the assembly line instead of boxes of stuffed animals we get container images. Whether or not our factory can be considered “secure” depends on how confident we are that those container images were not tampered with and are safe to use.

The term “Secure Software Factory” (SSF)  is in reference to a whitepaper released earlier this year by the CNCF Security TAG. This document provides a reference architecture for system architects to use for designing build pipelines which mitigate numerous supply chain security risks. These risks are outlined in detail in another whitepaper by the same group titled “Software Supply Chain Best Practices”.

We wrote a little bit about how Citi is building a Secure Software Factory in another blog post.

Building a Secure Software Factory is no small feat. There are many, many ways in which bad things can make their way into your release artifacts during the build process. From compromised credentials to compromised build nodes, attackers can get very creative in these environments.

Even considering an attacker was never able to modify a release artifact, these artifacts may contain components subject to future vulnerabilities. Do you know every version of every component inside your artifacts to mitigate this later? (see “SBOM”)

So one might ask: where do I start? The answer to this could really apply in any situation where there is an insurmountable amount of work to be done: start with low-hanging fruit.

(For those unfamiliar, “low-hanging fruit” is defined by Merriam-Webster as “the obvious or easy things that can be most readily done or dealt with in achieving success or making progress toward an objective”)

Image build: the low-hanging fruit

One of the lowest hanging fruits on this tree of doom is the image build. What tools are you using to build your images? Are you using Docker? Dockerfiles? Where are your base images coming from? (Remember, it's All About That Base Image!)

In containerized environments, the majority of vulnerable components will either surface during the building of the image or the running of the image. If you’re using a vulnerable base image, you could be doing everything else right and still get burned.

So, sure, you can start writing all sorts of fancy security policies which might quarantine insecure images if/when a vulnerability is detected, but how about starting to produce secure images from day 1?

Essential machines for the factory: melange and apko

Every factory has machines, right? Some machines are outdated and inefficient, while others are new and state-of-the-art.

We introduce you to 2 incredible machines that we believe belong in every Secure Software Factory: melange and apko.

These 2 tools in combination provide for a reproducible, declarative approach to building OCI images.

Did we mention, these machines are FREE!? (They’re open-source, of course!)


melange allows you to build APKs using declarative YAML pipelines. “APKs” refer to .apk packages compatible with apk (the package manager used by Alpine), similar to .deb or .rpm.


apko allows you to bundle a collection of APKs into an OCI image using a declarative YAML manifest.

Check out our blog post which originally introduced apko.

Nothing in your images but APKs

You’re probably familiar with Dockerfiles where you can “curl | bash” whatever you want. It depends on the context, but we are of the opinion that this type of thing is probably bad. Many Dockerfile-produced images also rely on a base image, which may or may not contain vulnerabilities.

OCI images are the required “shape” of things which run in containerized environments such as Kubernetes. However, there is no required “shape” things must be in order to make their way into an OCI image.

apko addresses this issue very simply: everything must be an APK.

This sounds neat in theory, but not everything you need is packaged as an APK, including your custom applications. This is where melange comes in.

Show me the code!

Alright then!

Currently melange and apk require apk-tools on Linux to run properly, so for the sake of simplicity, this example will use docker to run container images which contain these tools and all their dependencies.

The source for the following example can be found here. If you want to follow along, the remainder of the steps will assume you’ve done the following:

Let’s build an APK for a custom Go application, a simple HTTP server that responds “Hello World!”. Here’s the main.go:

Note: this directory should also include valid go.mod/go.sum files which reference the dependency on the gin framework.

To package this application up as APKs (one for each target architecture), we can define a melange.yaml as follows:

The 3 top-level sections in the file are:

  • package - metadata about the APK
  • environment - packages required for building this APK
  • pipeline - series of steps to run which ultimately populate ${{targets.destdir}} (the root of the APK)

As we will be signing our packages and repo index, first generate a temporary melange keypair:

Next, run the following to generate the APKs:

Note: the –privileged flag is required currently due to the use of bubblewrap internally which requires various Linux capabilities.

If melange ran successfully, you should end up with the following:

These APKs are now ready to be installed on an apk-based distro like Alpine.

This next part is a bit of a rough edge (see melange#96). In order for these APKs to be properly used, they must be part of an apk repo (even if just on the filesystem). Using the following command, we can generate a signed repo index for each architecture:

Now you should see the following (notice the new APKINDEX.tar.gz files):

Next, let’s take a look at our apko.yaml, which will bundle our custom APKs into OCI images:

Notice that we have defined a local repository at the path /work/packages. So we will mount the current working directory to  /work so that apko is able to find the hello-server package.

Run the following to build an OCI image with the tag factory-demo exported to output.tar:

A quick side note here: apko generated images are reproducible - run the same build with the same inputs and you will get bitwise identical output. This is something that is currently impossible to do with Docker. If you were to run the command above a second time using “output2.tar”, you should be able to verify these files are identical:

If the apko build was successful, you should be able to load the tarball into Docker:

Now we can run it!

Then in another terminal window, try to hit the server using curl:

That’s it! If you were to use apko publish to push the image instead, apko will automatically attach an SBOM containing all of the APKs included in the image! (This is probably grounds for a whole other blog post.)

But what if I don’t use Go?

That is no problem! APKs are general purpose that can be used to package really anything.

Just to prove that point, we put together duplicate demo apps that you could be used in place of the Go example, located in the same GitHub repo:

Just clone down the repo, enter the directory of the variant you wish to try, and you should be able to run the example commands from the previous section verbatim.

This is amazing!

You know what? We agree! We agree so much that we’ve started to put together a collection of GitHub Actions that will help automate all of these steps. Also, if you like melange and apko, throw ‘em a star on GitHub!

In a recent blog post, we introduced the new distroless project. All of the images produced by the distroless project are created using nothing but melange and apko!

So if you’re feeling like your software factory could use a bit of modernization, here’s a fantastic place for you to get started.

Show Comments