docker from scratch in

I have been using for a couple of years now on various projects – just a Makefile, but what a Makefile! I’ve recently added a fork that includes support for creating docker containers from scratch.

On MacOS you might want to run shortishly/docker-erlang which will give you erlang and docker packaged together in a shell (brew install docker-machine if you don’t already have it):

docker run \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -t \
       -i \
       --rm \
       shortishly/docker-erlang \

This is a short tutorial that assumes you are running on Linux, with both erlang and docker installed:

First lets create and move into a directory for this tutorial:

mkdir demo && cd demo

We need a copy of from the fork that I have created:

wget -q

Bootstrap a new project as you would do normally, including a release template:

make -f bootstrap
make bootstrap-rel

Also bootstrap this project to create a Docker container:

make bootstrap-docker

The “bootstrap-docker” target has created a Dockerfile for us:

FROM scratch


ENV BINDIR /erts-7.2.1/bin
ENV BOOT /releases/1/demo_release
ENV CONFIG /releases/1/sys.config
ENV ARGS_FILE /releases/1/vm.args


ENTRYPOINT exec ${BINDIR}/erlexec -boot_var /lib -boot ${BOOT} -noinput -config ${CONFIG} -args_file ${ARGS_FILE}

ADD _rel/demo_release/ /

Finally! Lets build the demo:


There should be lots of output at this point. At the end you should see:

===> Starting relx build process ...
===> Resolving OTP Applications from directories:
===> Resolved demo_release-1
===> Including Erts from /home/pmorgan/opt/erlang/18.2.1
===> release successfully created!
 GEN    docker-scratch-cp-dynamic-libs
 GEN    docker-scratch-cp-link-loader
 GEN    docker-scratch-cp-sh
 GEN    docker-strip-erts-binaries
 GEN    docker-build

A release has been successfully created “demo_release-1”, together with a bunch of new docker based targets and a new docker image (you will have a different sh256 output). You can check the output of docker images:

docker images

REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
demo_release               0.0.1               6440d984e0ef        3 minutes ago       22.02 MB

You can run the image with make docker-run:

make docker-run

 GEN    docker-rm
 GEN    docker-run

Again, you will have a different hash shown for the container that is now running – you can check that it is running with docker ps -a:

docker ps -a

CONTAINER ID        IMAGE                COMMAND                  CREATED              STATUS              PORTS               NAMES
b9deb1f7ef33        demo_release:0.0.1   "/bin/sh -c 'exec ${B"   About a minute ago   Up About a minute                       demo_release

You can use the shortcut make docker-logs to display the logs from your container:

make docker-logs

 GEN    docker-logs
heart_beat_kill_pid = 1

So with the help of we have been able to bootstrap an erlang project, building and running a docker container in about 10 shell commands.

erlang in docker from scratch

When packaging an application as a Docker container it is too easy to just be lazy and put FROM debian (other distributions are available, replace debian with your distribution of choice). For sure it is going to work, but you have just included dozens of libraries and binaries that your application just does not need. An image that could be tens of megabytes is now at least several hundred – we are building containers not virtual machines here!

One of the things I like about Go is that typical application binaries are small with no runtime dependencies. Fewer dependencies mean less patching and security headaches. The less friction in the CI build cycle, the better. Go achieves this by having statically linked applications meaning that just one binary is necessary in ADD, and they are typically built from scratch (etcd as a good example).

Erlang was designed to be embedded in telecoms equipment, so we must be able to package applications in Docker with a small footprint too?

An example repository containing a regular erlang application that is packaged in a docker container from scratch. You will need both erlang installed and docker service running preferably on a Linux environment. The release needs to be built on Linux to be able to run on Linux because we are going include the ERTS.

On MacOS you might want to run shortishly/docker-erlang which will give you erlang and docker packaged together in a shell (brew install docker-machine if you don’t already have it):

docker run \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -t \
       -i \
       --rm \
       shortishly/docker-erlang \

Clone and build the erlang-in-docker-from-scratch repository, which contains a minimal erlang application that builds a release into the _rel directory:

git clone eidfs
cd eidfs

At the end of make a standard erlang release for the eidfs application is now present in the _rel directory. To make it run inside a scratch container we need to include any runtime dependencies too. This is where mkimage comes in:


The ./bin/mkimage script copies in any dynamic libraries that ERTS needs to run the erlang release:

'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'
'/lib/x86_64-linux-gnu/' -> '_rel/eidfs/lib/x86_64-linux-gnu/'

It also copies /bin/sh so that we can run the release too. We can build a docker image for the release using the following command:

docker build \
       --build-arg REL_NAME=$(bin/release_name) \
       --build-arg ERTS_VSN=$(bin/system_version) \
       --pull=true \
       --no-cache=true \
       --force-rm=true \
       -t $(bin/release_name):$(bin/version) .

The $(bin/release_name), $(bin/system_version) and $(bin/version) are short escripts that respond with the release name, system ERTS version and the application version respectively.

Quite a lot of effort, what is the reward? Try docker images and look at the size of the resultant container:

REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
eidfs                      0.0.1               6748931f94e4        4 seconds ago       16.74 MB

We have a docker packaged erlang release in ~17MB. Lets run it!

docker run \
       --name $(bin/release_name) \
       -d \

Check the logs using docker logs $(bin/release_name) and you will see lots of application startup messages from SASL.

You might notice that the ENTRYPOINT used in the Dockerfile directly invokes erlexec. I have done this to reduce dependencies further so that the release, ERTS dynamic libraries, and /bin/bash only are present in the container.

FROM scratch
MAINTAINER Peter Morgan <>


ENV BINDIR /erts-${ERTS_VSN}/bin
ENV BOOT /releases/1/${REL_NAME}
ENV CONFIG /releases/${REL_VSN}/sys.config
ENV ARGS_FILE /releases/${REL_VSN}/vm.args


ENTRYPOINT exec ${BINDIR}/erlexec \
           -boot_var /lib \
           -boot ${BOOT} \
           -noinput \
           -config ${CONFIG} \
           -args_file ${ARGS_FILE}

ADD _rel/${REL_NAME}/ /

A needle in a…

Just about everyone is using Docker. As a developer it radically simplifies how I package my code. However, in a tiered architecture, how is my presentation layer load balanced over my API layer? Out of the box Docker linking is to a single container and doesn’t provide load balancing. I’d like to introduce Haystack which provides automatic service discovery and load balancing of HTTP or WebSocket based services composed on Docker containers from a single node to a Swarm.

High Level Architecture
High Level Architecture

Haystack does this by monitoring the event stream of a single node or Swarm, noticing when containers start or stop (gracefully or otherwise). On startup a container is registered in Haystack’s DNS and any HTTP or WebSocket endpoints are automatically added to its load balancing proxy. Later when that same container stops, Haystack automatically removes it from the load balancer dropping its entry from DNS. Containers are organised into groups, partitioning services into tiers which may then be load balanced over all nodes with an Overlay network.

Haystack is itself a docker container which may be run on a single node or on every node in a swarm providing load balancing that is local to the node, or to other nodes in the cluster. We can create a simple Haystack environment using Docker Machine as follows:

# create a new docker machine environment called "default"
docker-machine create --driver virtualbox default

# use that environment in this shell
eval $(docker-machine env default)

# start Haystack connecting it to the Docker daemon
docker run -e DOCKER_HOST=${DOCKER_HOST} \
  -e DOCKER_KEY="$(cat ${DOCKER_CERT_PATH}/key.pem)" \
  -e DOCKER_CERT="$(cat ${DOCKER_CERT_PATH}/cert.pem)" \
  -e SHELLY_AUTHORIZED_KEYS="$(cat ~/.ssh/authorized_keys)" \
  --name=haystack \
  --publish=53:53/udp \
  --publish=80:80 \
  --publish=8080:8080 \
  --publish=22022:22 \
  --detach \

Haystack has 3 blocks of environmental configuration in the above command:

  • Lines 8 through 10 using DOCKER_HOST, DOCKER_KEY and DOCKER_CERT are enabling Haystack to communicate with the Docker Daemon using TLS.
  • The line containing SHELLY_AUTHORIZED_KEYS is optional. It copies your public keys into the Haystack container so that you can SSH directly into Haystack to perform maintenance or run tracing if things aren’t working OK. If you’re familiar with the Erlang Shell you can do so right now with ssh -p 22022 $(docker-machine ip default).
  • Finally the DNS_NAMESERVERS tells Haystack where to delegate any unresolved names. In the above example we’re using Google’s Public DNS. You can replace these entries with your own DNS if you wish.

Lets start a couple of nginx servers to be our web tier connecting them to the Haystack DNS service:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --detach nginx

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --detach nginx

The --dns options are telling the Nginx docker container to resolve its DNS using the Haystack container.

Haystack maintains a DNS SRV record for each endpoint exposed by a container. For example, starting an Nginx container will automatically populate an appropriate service record into Haystack’s DNS using the IP address and port exposed by that container. In addition, a DNS A record for that service is created which points to Haystack’s Load Balancer.

# nslookup -query=srv $(docker-machine ip default) service = 100 100 80 c9pcjp7.containers.haystack.	service = 100 100 80 c5162p7.containers.haystack.

# dig -query=a $(docker-machine ip default) 100	IN	A

Haystack has automatically created a DNS entry called which is pointing to its own internal load balancing proxy. We can curl that address and the HTTP or WebSocket requests will be load balanced over the available Nginx instances:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty --interactive --rm fedora /bin/bash

[root@a268fc00929d /]# curl
<!DOCTYPE html>
<title>Welcome to nginx!</title>

The DNS hierarchy created by Haystack is as follows:

Haystack DNS Hierarchy

Services that are available to be load balanced are published under the services.haystack namespace (for example, Haystack internally uses SRV records for each service instance, which Munchausen (the load balancing proxy) also references. Each docker container is also registered with a private Haystack name under the containers.haystack namespace.