Open Source

sigstore, the local way

Thomas Strömberg, Director of Security
February 16, 2022
copied

If you've been following the Chainguard blog, you might ask yourself: how do I run the open-source sigstore stack on my machine?

While sigstore is often deployed using Kubernetes, it is flexible enough to run nearly anywhere: from a Raspberry Pi to an IBM mainframe. This article will demonstrate how to build the sigstore stack (cosign, rekor, fulcio) on your machine and use it to sign and verify container signatures without ever leaving localhost.

Prerequisites

Consult the documentation for your favorite package manager to install:

  • The Go Programming Language (v1.16 or higher: confirm by running go version )
  • The MariaDB SQL server
  • The Git distributed version control system
  • OpenSC for managing PKCS#11 security tokens
  • SoftHSM for implementing a PKCS#11 storage interface

Examples:

  • The Go Programming Language (v1.16 or higher: confirm by running go version )
  • The MariaDB SQL server
  • The Git distributed version control system
  • OpenSC for managing PKCS#11 security tokens
  • SoftHSM for implementing a PKCS#11 storage interface

These dependencies may also be downloaded and installed from their respective websites.

Level I: Keyed signing with a local registry

First, we will use cosign, sigstore's container-signing tool, to sign a locally published container using a locally maintained key pair. Here are the commands we will execute, along with where they will access data from:

1.1: Running a local registry

sigstore can sign containers stored within any container registry. To keep our demonstration local, we'll install a simple registry using Go. Open a terminal and run:

-- CODE language-bash -- go install github.com/google/go-containerregistry/cmd/registry@latest

As we begin launching several services into the foreground, now is a great time to begin a shell multiplexer such as tmux or screen, or at least a terminal that supports tabs. Start the registry service:

-- CODE language-bash -- $HOME/go/bin/registry

The terminal will now quietly hang until a request arrives. Start a new terminal session, and let's move on!

1.2: Pushing an unsigned image to the local registry

So that we have a target container to sign, we will now build a sample container and upload it to the local registry. First, install the ko container builder:

-- CODE language-bash -- go install github.com/google/ko@latest

Download rekor, sigstore's tamper-resistant ledger software:

-- CODE language-bash -- mkdir -p $HOME/sigstore-local/src cd $HOME/sigstore-local/src git clone https://github.com/sigstore/rekor.git

Build and push an image containing the rekor CLI to our local registry:

-- CODE language-bash -- cd $HOME/sigstore-local/src/rekor/cmd KO_DOCKER_REPO=localhost:1338/demo $HOME/go/bin/ko publish ./rekor-cli

1.3: Keyed-signing with cosign

Now we get to use cosign, sigstore's container-signing tool, to sign the container we just published. Install the latest cosign release:

-- CODE language-bash -- go install github.com/sigstore/cosign/cmd/cosign@latest

Create a local key pair using any password. For simplicity, I suggest an empty one:

-- CODE language-bash -- cd $HOME/sigstore-local $HOME/go/bin/cosign generate-key-pair

Sgn the published container using the local private key:

-- CODE language-bash -- $HOME/go/bin/cosign sign --key cosign.key \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest

Use cosign to verify that the published container matches the local public key:

-- CODE language-bash -- $HOME/go/bin/cosign verify --key cosign.pub \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

The output for a successful verification will look like this:

-- CODE language-bash -- Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key - Any certificates were verified against the Fulcio roots. [{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:3a46c2e44bfe8ea0231af6ab2f7adebd0bab4a892929b307c0b48d6958863a4d"},"type":"cosign container image signature"},"optional":null}]

Congratulations! You have just signed your first container! To sign other artifacts such as binaries, see Working with other artifacts.

Level II: Certificate Transparency with Rekor

So far, verification has relied on a single mutable source of truth: the container registry. With Rekor, we will introduce a second immutable source of truth to the system:

2.1: Creating a database backend with MariaDB

While Sigstore can use multiple database backends, this tutorial uses MariaDB. Once you've installed the prerequisites, run the following to start the database up locally in a locked-down manner:

  • Arch Linux: sudo mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql; sudo systemctl start mariadb && sudo mysql_secure_installation
  • Debian|Ubuntu: sudo mysql_secure_installation
  • Fedora: sudo systemctl start mariadb && sudo mysql_secure_installation
  • FreeBSD: sudo sudo service mysql-server start && sudo mysql_secure_installation
  • macOS: brew services start mariadb && sudo mysql_secure_installation
  • OpenBSD: doas mysql_install_db && doas rcctl start mysqld && doas mysql_secure_installation

Follow the prompts from mysql_install_db, answering N to change the root password, and Y to everything else. Afterward, run the database creation script:

-- CODE language-bash -- cd $HOME/sigstore-local/src/rekor/scripts sudo sh -x createdb.sh

2.2: Installing Trillian

Trillian provides a tamper-proof append-only log based on Merkle Trees using a gRPC API. Trillian stores its records in the MariaDB database we previously created. Install Trillian:

-- CODE language-bash -- go install github.com/google/trillian/cmd/trillian_log_server@latest go install github.com/google/trillian/cmd/trillian_log_signer@latest go install github.com/google/trillian/cmd/createtree@latest

Start the log_server, which provides the Trillian "personality" API, used by Rekor, and the Certificate Transparency frontend.

-- CODE language-bash -- $HOME/go/bin/trillian_log_server --logtostderr \ -http_endpoint=localhost:8090 -rpc_endpoint=localhost:8091

Start the log signer, which periodically checks the database and sequences data into a Merkle tree:

-- CODE language-bash -- $HOME/go/bin/trillian_log_signer \ --logtostderr --force_master --http_endpoint=localhost:8190 \ --rpc_endpoint=localhost:8191

The Trillian system is multi-tenant and can support multiple independent Merkle trees. Run this command to send a gRPC request to create a tree and save the log_id for future use:

-- CODE language-bash -- $HOME/go/bin/createtree --admin_server localhost:8091 \ | tee $HOME/sigstore-local/trillian.log_id

2.3: Installing Rekor

The Rekor project provides a restful API-based server for validation and a transparency log for storage. Install it from source:

-- CODE language-bash -- cd $HOME/sigstore-local/src/rekor go install ./cmd/rekor-cli ./cmd/rekor-server

Start rekor:

-- CODE language-bash -- $HOME/go/bin/rekor-server serve --trillian_log_server.port=8091 \ --enable_retrieve_api=false

Upload a test artifact to verify that Rekor is functioning correctly:

-- CODE language-bash -- cd $HOME/sigstore-local/src/rekor $HOME/go/bin/rekor-cli upload --artifact tests/test_file.txt \ --public-key tests/test_public_key.key \ --signature tests/test_file.sig \ --rekor_server http://localhost:3000

2.4: Verifiable signing with Cosign & Rekor

Upload a signature for our image, using the local key pair we created in step 1.3.

-- CODE language-bash -- COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign \ --key $HOME/sigstore-local/cosign.key \ --rekor-url=http://localhost:3000 \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Verify the container against the mutable OCI attestation and the immutable Rekor record:

-- CODE language-bash -- COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign verify \ --key $HOME/sigstore-local/cosign.pub \ --rekor-url=http://localhost:3000 \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Success looks like this:

-- CODE language-bash -- Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest -- The following checks were performed on each of these signatures: - The cosign claims were validated - The claims were present in the transparency log - The signatures were integrated into the transparency log when the certificate was valid - The signatures were verified against the specified public key - Any certificates were verified against the Fulcio roots. [{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:35b25714b56211d548b97a858a1485b254228fe9889607246e96ed03ed77017d"},"type":"cosign container image signature"},"optional":{"Bundle":{"SignedEntryTimestamp":"MEUCIG...yoIY=","Payload":{"body":"...","integratedTime":1643917737,"logIndex":1,"logID":"4d2e4...97291"}}}}]

🍦 Take an ice cream break if you got this far. You earned it!

Level III:  Keyless signing with Fulcio

Fulcio is a sigstore component that issues code-signing certificates based on an authenticated OpenID Connect identity. These certificates are short-lived, lasting only 20 minutes!

Cosign has experimental support for using Fulcio to generate signing certificates, saving users the headache of managing certificates. As our goal is to run everything locally, we have to stand up substantially more architecture to provide secure keyless signing:

3.1: Install Fulcio

As we're going to use some experimental features in this tutorial, you will need to install Fulcio from HEAD:

-- CODE language-bash -- cd $HOME/sigstore-local/src git clone https://github.com/sigstore/fulcio.git cd fulcio go install .

3.2: Configure SoftHSM

SoftHSM implements a cryptographic store accessible through a PKCS #11 interface. You can use it to explore PKCS #11 without having a Hardware Security Module. For this demo, we will configure sigstore to reference tokens in $HOME/sigstore-local/tokens :

Create your first HSM token, setting both PINs to 2324.

-- CODE language-bash -- softhsm2-util --init-token --slot 0 --label fulcio

3.3: Create a CA certificate with OpenSC

Many tools need to be informed about the SoftHSM installation location, which is different in each operating system. Run this command to set and reveal the library location:

-- CODE language-bash -- export HSM=$(find /usr/local/lib /usr/lib /usr/lib64 \ /opt/homebrew -name libsofthsm2.so | head -n1) file $HSM

Create a configuration file for the pkcs11 crypto library using the user PIN code you specified in the previous step:

-- CODE language-bash -- mkdir -p $HOME/sigstore-local/config echo "{ \"Path\": \"$HSM\", \"TokenLabel\": \"fulcio\", \"Pin\": \"2324\" }" > $HOME/sigstore-local/config/crypto11.conf

Generate a new key pair that is stored directly in the HSM. When prompted for a PIN, use 2324:

-- CODE language-bash -- SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf pkcs11-tool \ --login --login-type user --keypairgen --id 1 --label PKCS11CA \ --key-type EC:secp384r1 --module=$HSM

Create a local CA root certificate:

-- CODE language-bash -- cd $HOME/sigstore-local SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf $HOME/go/bin/fulcio \ createca --org=acme --country=USA --locality=Anytown \ --province=AnyPlace --postal-code=ABCDEF \ --street-address="123 Main St" --hsm-caroot-id 1 --out ca-root.pem

3.4: Install the Certificate Transparency Frontend

The ct_server is an RFC6962-compliant certificate transparency log that stores the code-signing certificates issued by Fulcio.

-- CODE language-bash -- go install github.com/google/certificate-transparency-go/trillian/ctfe/ct_server@latest

Next, create a private key for the front end to use for signing certificates. For the password, I suggest using 2324 again, but you may use anything 4-characters or longer:

-- CODE language-bash -- cd $HOME/sigstore-local openssl ecparam -genkey -name prime256v1 -noout -out ct_unenc.key openssl ec -in ct_unenc.key -out ct_private.pem -des openssl ec -in ct_unenc.key -out ct_public.pem -pubout -des rm ct_unenc.key

Store the password as a shell variable:

-- CODE language-bash -- export PASS= <the password you just used>

Look up the Trillian log ID we previously created and set the LOG_ID variable to the resulting value:

-- CODE language-bash -- Look up the Trillian log ID we previously created and set the LOG_ID variable to the resulting value:

Populate the Certificate Transparency configuration file:

-- CODE language-bash -- printf "config { log_id: $LOG_ID prefix: \"sigstore\" roots_pem_file: \"$HOME/sigstore-local/ca-root.pem\" private_key: { [type.googleapis.com/keyspb.PEMKeyFile] { path: \"$HOME/sigstore-local/ct_private.pem\" password: \"$PASS\" } } }" | tee $HOME/sigstore-local/ct.cfg

Start the certificate transparency server:

-- CODE language-bash -- $HOME/go/bin/ct_server -logtostderr \ -log_config $HOME/sigstore-local/ct.cfg \ -log_rpc_server localhost:8091 \ -http_endpoint 0.0.0.0:6105

3.5: Installing Dex for OpenID authentication

Dex is a federated OpenID Connect Provider, connecting OpenID identities from multiple providers. We are going to use Dex to provide GitHub authentication. To build it from source (Note: BSD users should use gmake instead of make):

-- CODE language-bash -- cd $HOME/sigstore-local/src git clone https://github.com/dexidp/dex.git cd dex make build cp bin/dex $HOME/go/bin

For this demonstration, we'll use GitHub as an OpenID provider. Visit GitHub: Register a new OAuth Application, and fill in the form accordingly:

💡
Application Name: My Local Sigstore Adventure
Homepage URL: http://localhost/
Authorization callback URL: http://localhost:5556/callback

When you click Register Application, it will output a client ID. Save it to your environment:

-- CODE language-bash -- export GI_ID= <your id>

Click the Generate a new client secret button, and copy the long alphanumeric string it emits into your environment:

-- CODE language-bash -- export GI_SECRET=<your client secret>

Populate the Dex configuration:

-- CODE language-bash -- printf "issuer: http://localhost:5556 storage: type: sqlite3 config: file: ./dex.db web: http: 127.0.0.1:5556 frontend: issuer: sigstore oauth2: responseTypes: [ "code" ] staticClients: - id: sigstore public: true name: sigstore connectors: - type: github id: github-sigstore-test name: GitHub config: clientID: $GI_ID clientSecret: $GI_SECRET redirectURI: http://localhost:5556/callback " | tee $HOME/sigstore-local/dex-config.yaml

Start dex:

-- CODE language-bash -- $HOME/go/bin/dex serve $HOME/sigstore-local/dex-config.yaml

3.6: Setting up Fulcio for keyless signatures

Populate the Fulcio configuration:

-- CODE language-bash -- printf '{ "OIDCIssuers": { "http://localhost:5556": { "IssuerURL": "http://localhost:5556", "ClientID": "sigstore", "Type": "email" } } }' > $HOME/sigstore-local/config/fulcio.json

Start Fulcio:

-- CODE language-bash -- cd $HOME/sigstore-local SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf \ $HOME/go/bin/fulcio serve --config-path=config/fulcio.json \ --ca=pkcs11ca --hsm-caroot-id=1 \ --ct-log-url=http://localhost:6105/sigstore \ --host=127.0.0.1 --port=5000

3.7: Local Keyless Signing

Now it is time for the finale: keyless signing! In an internet-connected environment, the command-line to sign a container is relatively simple:

-- CODE language-bash -- COSIGN_EXPERIMENTAL=1 cosign sign <image location>

However, since we will be using our local environment, we need to override the public-key certificate used to verify identities and the endpoint locations. Run this command to sign the image we pushed locally:

-- CODE language-bash -- SIGSTORE_CT_LOG_PUBLIC_KEY_FILE=$HOME/sigstore-local/ct_public.pem \ COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign \ --oidc-issuer=http://localhost:5556 \ --fulcio-url=http://127.0.0.1:5000 \ --rekor-url=http://localhost:3000 \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

When you run that command, your browser will open to the local Dex instance, which will prompt you to authenticate using GitHub. Afterward, verify the certificate:

-- CODE language-bash -- SIGSTORE_ROOT_FILE=$HOME/sigstore-local/ca-root.pem \ COSIGN_EXPERIMENTAL=1 \ $HOME/go/bin/cosign verify \ --rekor-url=http://localhost:3000 \ localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9

Among other things, the output will include the OIDC issuer (our local Dex URL) and the e-mail address you authenticated to Github with. Here is an example:

-- CODE language-bash -- Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest -- The following checks were performed on each of these signatures: - The cosign claims were validated - The claims were present in the transparency log - The signatures were integrated into the transparency log when the certificate was valid - Any certificates were verified against the Fulcio roots. [{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:dae1e7cdb03fc6f16e3a48111634f0c5fc28752229da2f96a6a4d03f60d6 d609"},"type":"cosign container image signature"}, "optional":{"Bundle":{"SignedEntryTimestamp":"MEUCI...Og4=","Payload":{"body":"...","integratedTime":1644977812,"logIndex":3,"logID":"9d0dd...a86d"}}, "Issuer":"http://localhost:5556","Subject":"blog@chainguard.dev"}}]

😊
Pat yourself on the back! You made it through the tutorial!

Where to next?

If you encounter any problems or would like to learn more about sigstore, see:

Happy artifact signing, everyone!

Related articles

Ready to lock down your supply chain?

Talk to our customer obsessed, community-driven team.