Building images introduces many challenges, one of them is safety of persisted data. How to avoid publishing sensitive data, that should remain private?

I don’t speak about code itself, because if we build images for internal usage, we must ensure safe storage and usage, which is totally different story. Imagine though we build publicly available tool that helps in day by day work, and its code is also public (e.g. composer). So where is the place for secrets in the release process?

What are secrets?

Secrets are data, as far as its name suggests, that should remain only for ourselves. These can be passwords, tokens or certificates used for integrating with external services. It was already possible in Docker to use such secrets in Swarm stacks based on docker compose. Thanks to that it’s really easy to pass sensitive data to the application, but only in runtime — secrets were not available during build, aren’t persisted inside build image (hence it’s not possible to get there from build layers’ cache), in different words: can be used only after passing them to running container.

Sensitive data in a build process

Mentioned #Composer and other package managers, during image build process are used for resolving and downloading dependencies. In case of Open Source projects most often these dependencies are also publicly available on GitHub, Gitlab or other. But those providers secure themselves from improper usage, so for mass downloading you probably need to authorise, hence you need tokens on build stage.

While building images that use packages from external registries (Composer, NPM, PYPI), we may need to integrate with private registry (Private Packagist, Artifactory, Verdaccio). While the presence of the packages’ code inside the image does not have to be a problem for us, we would not necessarily want to share the authentication data with others 😉

Anyway, it’s not only about packages — we may want to download some artifacts: translations’ definition, static files… And these files may be stored in external systems integrated through API, in which we need authenticate too.

Secrets during build

Few days ago in docker/compose repository, long-awaited feature (issue from 2018!) was implemented and merged into main v2 branch — secrets during build 🎉. When it gets released, it can be used like:

services:
  ssh:
    image: build-test-secret
    build:
      context: .
      secrets:
        - mysecret

secrets:
  mysecret:
    file: ./secret.txt

FROM alpine

RUN echo "foo" > /tmp/expected
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret > /tmp/actual
RUN diff /tmp/expected /tmp/actual

Having such Dockerfile and #Docker Compose stack we can safely pass sensitive data into Docker’s build context. Of course storing it into the file is not a great idea, this example only shows how to pass and use the secret 😉. If we use it correctly, it won’t be available in built image and within its build history (image layers can be browsed).

It will enable consistent build process using secrets both for #CI and for local development.

Now we only need to wait for new Docker Compose release 😁