Featured

Docker Service Discovery and Load Balancing with Haystack

Haystack provides service discovery and automatic HTTP load balancing in a Docker environment. It uses its own DNS to provide service discovery and dynamically updates its embedded load balancer as services are started or stopped.

Haystack solves two problems:

  • Service Discovery – Haystack automatically manages service registration within DNS by monitoring the Docker container lifecycle. Haystack introspects container metadata exposing available services within its own DNS. Haystack automatically manages the registration of services into a common DNS for discovery by other services.
  • Load balancing of a service as it is scaled up or down to meet demand – Haystack registers DNS service names that are load balanced over the available instances as the service is scaled up or down to meet demand. Each service name is registered in Haystack’s own DNS and Haystack manages the load balancing of the containers providing that service, from one to thousands and back again – dynamically as demand rises or falls.

This post walks through a demo of some of capabilities that Haystack provides using a simple HTTP based micro service. The service accepts HTTP GET requests and responds with the hostname of the container and the HTTP path used in the request.

To start a number of services to demonstrate load balancing in Haystack:

for i in {1..5}; do
     docker run \
     --name demo-$(printf %03d $i) \
     -d shortishly/haystack_demo;
done

Start Haystack – replace 172.16.1.218 with the location of your Docker Engine that is already listening on a tcp port.

docker run \
    -p 8080:80 \
    -e DOCKER_HOST=tcp://172.16.1.218:2375 \
    -d \
    --name haystack \
    shortishly/haystack

You should now have Haystack and 5 demo micro services running within Docker:

# docker ps --format="{{.Names}}"

haystack
demo-001
demo-002
demo-003
demo-004
demo-005

Now start a busybox that uses Haystack for DNS resolution as follows:

  docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty \
  --interactive \
  --rm busybox /bin/sh

In the busybox shell:

wget \
   -q \
   -O /dev/stdout \
    http://demo.haystack_demo.services.haystack/this/is/a/demo

Haystack has registered demo.haystack_demo.services.haystack in
its own DNS service, and is load balancing any
request randomly to one of the demo-001, demo-002,
demo-003, demo-003, demo-004 or demo-005
containers.

If you make a number of wget requests to the same URL you will
get responses from the different containers at random:

# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
d617596e70da: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
96c6c6f27f03: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
296b5208edf9: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
1166e110e70d: /load/balancing
# wget -q -O /dev/stdout http://demo.haystack_demo.services.haystack/load/balancing
9909343b937a: /load/balancing

You can verify which services are available in Haystack by curling /api/info:

curl -s http://$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack)/api/info|python -m json.tool

Use you bowser to visit the Haystack UI running on your Docker host:

http://YOUR_DOCKER_HOST:8080/ui

Automatic Load Balancing Of Spring Boot Micro Services In Docker With Haystack

In Spring Boot Micro Services In Docker we saw how to create a Docker container from a Spring Boot Micro Service. In this article we shall look at load balancing the service using Haystack. Haystack is a DNS based load balancer that is integrated with the Docker API automatically creating service groups as containers stop and start.

Haystack monitors the Docker API using a tcp socket. In this case the Docker API is on port 2375 on the Docker host.

docker run \
  -e DOCKER_HOST=tcp://172.16.1.218:2375 \
  --name haystack \
   --detach \
   shortishly/haystack

Build a gsa image following the instructions in this article, starting 3 instances of our Spring Boot Micro Service:

docker run --name srv-001 -d gsa
docker run --name srv-002 -d gsa
docker run --name srv-003 -d gsa

We can confirm that we have 3 gsa services by running docker ps as follows:

$ docker ps -a --format="{{.Names}}"

srv-003
srv-002
srv-001

Haystack automatically creates a service group srv.gsa.services.haystack in DNS. Starting a new gsa container will automatically add it to the service group. Stopping a gsa container automatically will remove it from the service group.

Startup a busy box instance using Haystack’s embedded DNS:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty \
  --interactive \
  --rm busybox /bin/sh

Lookup srv.gsa.services.haystack in DNS:

nslookup srv.gsa.services.haystack

Server:    172.17.0.7
Address 1: 172.17.0.7 cb1i9a6.containers.haystack

Name:      srv.gsa.services.haystack
Address 1: 172.17.0.7 cb1i9a6.containers.haystack

Note that srv.gsa.services.haystack is actually pointing to the Haystack container. This is because Haystack acts as a proxy to HTTP requests, automatically load balancing requests randomly over members of the service group. Issue a wget to the service group and the requests will be load balanced randomly over the members:

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":2,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":1,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":3,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":2,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":4,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":3,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":4,"content":"Hello, Stranger!"}

# wget -q -O /dev/stdout http://srv.gsa.services.haystack/hello-world
{"id":5,"content":"Hello, Stranger!"}

Congratulations! You now have a Spring Boot Micro Service Docker container that is automatically being load balanced by Haystack. You can add or remove further gsa services and Haystack will automatically update the service pool.

Spring Boot Micro Services In Docker

Building a RESTful Web Service with Spring Boot Actuator is a great example of building a very simple micro service using Spring Boot Actuator. In this article we will go on to package that service into a Docker container.

Firstly we need to clone the source code from the original article:

$ git clone https://github.com/spring-guides/gs-actuator-service.git
$ cd gs-actuator-service/complete/

Build it using gradle:

$ ./gradlew clean assemble

Or with maven:

$ mvn package -DskipTests=true

We need to create a Dockerfile that we will use to build a docker image of the service as follows:

FROM java:8
EXPOSE 80
COPY target/gs-actuator-service-0.1.0.jar /
ENTRYPOINT ["java", \
           "-Djava.security.egd=file:/dev/urandom", \
           "-jar", \
           "gs-actuator-service-0.1.0.jar", \
           "--server.port=80"]

A couple of notes on the Dockerfile:

  • We are using the official Java 8 docker image as our base and copying our jar into the root directory of the image. If you used gradle instead of maven replace target/gs-actuator-service-0.1.0.jar with build/libs/gs-actuator-service-0.1.0.jar
  • We are using /dev/urandom as explained by this article to avoid issues in slow Tomcat startup when there is insufficient entropy (also the reason why we skipped the tests when building above)
  • We expose the service on port 80. When managing more than a couple of containers it is much simpler to just use standard ports.

Using your favourite editor put the Dockerfile in the complete directory, and build a docker image as follows:

docker build -t gsa .

Our image has been created an has been tagged as ‘gsa’:

$ docker images
REPOSITORY TAG    IMAGE ID     CREATED       SIZE
gsa        latest 6acbba251766 4 seconds ago 683.1 MB

We can run the image using docker run as follows:

docker run --name gsa -d gsa

We can find the IP address being used by the gsa container by running docker inspect:

$ docker inspect \
    --format={{.NetworkSettings.IPAddress}} gsa

172.17.0.7

In this case our ‘gsa’ container has been given 172.17.0.7 as its IP address. We can make a HTTP request for /hello-world from our new service with the following:

curl 172.17.0.7/hello-world

Be sure to replace IP address 172.17.0.7, with the output that your docker host gives you on docker inspect.

In response you will get something like:

{"id":1,"content":"Hello, Stranger!"}

Congratulations! You now have a Spring Boot Actuator Micro Service running inside a Docker container.

Tansu is a distributed key/value and lock store

Tansu is a distributed key value store designed to maintain configuration and other data that must be highly available. It uses the Raft Consensus algorithm for leadership election and distribution of state amongst its members. By default node discovery is via mDNS and will automatically form a mesh of nodes sharing the same environment.

Features

Tansu has a REST interface to set, get or delete the value represented by a key. It also provides a HTTP Server Sent Event Stream
of changes to the store.

Tansu provides REST interface for simple Check And Set (CAS) operations.

Tansu provides test and set operations that can be used to operate locks through a simple REST based HTTP Server Sent Event Stream interface.

Quick Start

Tansu is packaged as a Docker container. To start a local 5 node cluster is as simple as:

for i in {1..5}; do
    docker run \
        --name tansu-$(printf %03d $i) \
        -d shortishly/tansu;
done

Tansu uses mDNS by default to discover other nodes and automatically forms a cluster. API requests can be made to any discovered node, which are internally routed to the appropriate node depending on the request type.

To demonstrate this, lets use the shell to randomly pick a node from our new cluster:

RANDOM_IP=$(docker inspect \
            --format={{.NetworkSettings.IPAddress}} \
            tansu-$(printf %03d $[1 + $[RANDOM % 5]]))

All key/value API operations are under the ‘/api/keys/…’ URL. We can create a stream of changes to a key (or a hierarchy of keys) before that key exists as follows:

curl \
    -i \
    -s \
    "http://${RANDOM_IP}/api/keys/hello?stream=true&children=true"

The key space in Tansu is a directory structure separated with ‘/’ characters. Any change to the key ‘hello’ will be reported in the above stream, and also any change in subdirectory below ‘hello’ will also be reported.

Leaving the stream curl running, in another shell lets assign the value “world” to the key “hello”:

curl \
    -X PUT \
    -i \
    -s \
    http://${RANDOM_IP}/api/keys/hello \
    -d value=world

Back in our stream, it will contain a ‘create’ notification:

id: 1
event: create
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
  "tansu":{
    "content_type":"text/plain",
    "created":1,
    "parent":"/",
    "updated":1}},
  "value":"world"}

Or a key that is below ‘hello’:

curl \
  -X PUT \
  -i \
  -s \
   http://${RANDOM_IP}/api/keys/hello/joe \
  -d value=mike

The stream will now contain a `create` notification:

id: 2
event: create
data: {
  "category":"user",
  "key":"/hello/joe",
  "metadata":{
    "tansu":{
      "content_type":"text/plain",
      "created":2,
      "parent":"/hello",
      "updated":2}},
  "value":"mike"}

In the above case Tansu will assume that the value has the ‘text/plain’ content type (as the value from a form url encoded body). Other content types (in particular JSON) are also supported:

curl \
  -X PUT \
  -H "Content-Type: application/json" \
  -i http://${RANDOM_IP}/api/keys/hello \
  --data-binary '{"stuff": true}'

With an update in the stream:

id: 3
event: set
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":1,
      "parent":"/",
      "updated":3}},
  "previous":"world",
  "value":{"stuff":true}}

GET

The current value of a key can be obtained simply by issuing a GET on that key:

curl \
  -i \
  -s \
  http://${RANDOM_IP}/api/keys/hello

{"stuff": true}

DELETE

Similarly a key is removed by issuing a DELETE request:

curl \
  -i \
  -X DELETE \
  http://${RANDOM_IP}/api/keys/hello

The stream will now contain a delete notification:

id: 5
event: delete
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":1,
      "parent":"/",
      "updated":5}},
  "value":{"stuff":true}}

TTL

A value can also be given a time to live by supplying a TTL header:

curl \
  -X PUT \
  -H "Content-Type: application/json" \
  -H "ttl: 10" \
  -i \
  http://${RANDOM_IP}/api/keys/hello \
  --data-binary '{"ephemeral": true}'

The event stream will contain details of the `create` together with a TTL
attribute:

id: 6
event: create
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":6,
      "parent":"/",
      "ttl":10,
      "updated":6}},
  "value":{"ephemeral":true}}

Ten seconds later when the time to live has expired:

id: 7
event: delete
data: {
  "category":"user",
  "key":"/hello",
  "metadata":{
    "tansu":{
      "content_type":"application/json",
      "created":6,
      "parent":"/",
      "ttl":0,
      "updated":7}},
  "value":{"ephemeral":true}}

Building Erlang/OTP docker releases with gitlab CI

Gitlab has awesome integrated support for CI. Building your next Erlang/OTP release into a Docker image could be as simple as committing a .gitlab-ci.yml to your project.

To run Gitlab in Docker locally follow these instructions. The command that I use is:

docker run --detach \
    --hostname YOUR_GITLAB_HOSTNAME \
    --publish 443:443 --publish 80:80 --publish 22:22 \
    --name gitlab \
    --restart always \
    --volume /data/gitlab/config:/etc/gitlab:Z \
    --volume /data/gitlab/logs:/var/log/gitlab:Z \
    --volume /data/gitlab/data:/var/opt/gitlab:Z \
    --volume /data/gitlab/logs/reconfigure:/var/log/gitlab/reconfigure:Z \
    gitlab/gitlab-ce:latest

My docker host is running Fedora with SE linux enabled, the “Z” on the volume mounts does some labelling magic making docker and SE happy together.

Our next step is to start a gitlab-runner to orchestrate our builds, which also can run as a Docker container:

docker run -d --name gitlab-runner --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock:Z \
  -v /data/gitlab-runner/config:/etc/gitlab-runner:Z \
  gitlab/gitlab-runner:latest

Note that I use /data as the host volume mount – replace with your preferred local mount point. We will be editing the configuration created in /data/gitlab-runner/config in a moment. We need to register this runner with gitlab, as follows:

docker exec -it gitlab-runner gitlab-runner register \
  --url "http://YOUR_GITLAB_HOSTNAME/ci" \
  --registration-token "YOUR_GITLAB_TOKEN" \
  --description "docker-ruby-2.1" \
  --executor "docker" \
  --docker-image ruby:2.1

Finally we need to volume mount /var/run/docker.sock into the runner so that we can run docker within the runner. Modify /data/gitlab-runner/config/config.toml so that the volumes include:

  volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]

Restart the runner with:

docker stop gitlab-runner && docker start gitlab-runner

Phew! You now have a gitlab runner setup ready to build some software! Gitlab will build a project providing it has a .gitlab-ci.yml present in the root directory of that project:

image: shortishly/docker-erlang

stages:
  - build

app:
  stage: build
  script:
    - make
    - docker login -e $DOCKER_EMAIL -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker tag $(bin/release_name):$(bin/version) $DOCKER_USERNAME/$(bin/app):$(bin/version)
    - docker push $DOCKER_USERNAME/$(bin/app):$(bin/version)
  only:
    - master

The image shortishly/docker-erlang is a container (built from this Dockerfile) that is from the official erlang docker image, but also has Docker installed so that we can build, tag and push.

The bin/release_name, bin/version are bin/app helper scripts from this project. The above is an erlang.mk based build using a fork that automatically creates a Docker container from Erlang/OTP release. To use in your own project replace the existing erlang.mk by using:

wget -q https://github.com/shortishly/erlang.mk/raw/master/erlang.mk

Commit the new erlang.mk and .gitlab-ci.yml and check Gitlab to see whether the build is progressing. Assuming everything is OK a new docker image will be pushed with your release.

docker from scratch in erlang.mk

I have been using erlang.mk for a couple of years now on various projects – just a Makefile, but what a Makefile! I’ve recently added a fork that includes support for creating docker containers from scratch.

On MacOS you might want to run shortishly/docker-erlang which will give you erlang and docker packaged together in a shell (brew install docker-machine if you don’t already have it):

docker run \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -t \
       -i \
       --rm \
       shortishly/docker-erlang \
       /bin/bash

This is a short tutorial that assumes you are running on Linux, with both erlang and docker installed:

First lets create and move into a directory for this tutorial:

mkdir demo && cd demo

We need a copy of erlang.mk from the fork that I have created:

wget -q https://github.com/shortishly/erlang.mk/raw/master/erlang.mk

Bootstrap a new erlang.mk project as you would do normally, including a release template:

make -f erlang.mk bootstrap
make bootstrap-rel

Also bootstrap this project to create a Docker container:

make bootstrap-docker

The “bootstrap-docker” target has created a Dockerfile for us:

FROM scratch

ARG REL_NAME
ARG REL_VSN=1
ARG ERTS_VSN

ENV BINDIR /erts-7.2.1/bin
ENV BOOT /releases/1/demo_release
ENV CONFIG /releases/1/sys.config
ENV ARGS_FILE /releases/1/vm.args

ENV TZ=GMT

ENTRYPOINT exec ${BINDIR}/erlexec -boot_var /lib -boot ${BOOT} -noinput -config ${CONFIG} -args_file ${ARGS_FILE}

ADD _rel/demo_release/ /

Finally! Lets build the demo:

make

There should be lots of output at this point. At the end you should see:

===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/pmorgan/src/git/demo/ebin
          /home/pmorgan/opt/erlang/18.2.1/lib
          /home/pmorgan/src/git/demo/apps
          /home/pmorgan/src/git/demo/deps
===> Resolved demo_release-1
===> Including Erts from /home/pmorgan/opt/erlang/18.2.1
===> release successfully created!
 GEN    docker-scratch-cp-dynamic-libs
 GEN    docker-scratch-cp-link-loader
 GEN    docker-scratch-cp-sh
 GEN    docker-strip-erts-binaries
 GEN    docker-build
sha256:6440d984e0efb5a7f10168e805cdfcf47f670b03ce953427ba2613b177072d8c

A release has been successfully created “demo_release-1”, together with a bunch of new docker based targets and a new docker image (you will have a different sh256 output). You can check the output of docker images:

docker images

REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
demo_release               0.0.1               6440d984e0ef        3 minutes ago       22.02 MB

You can run the image with make docker-run:

make docker-run

 GEN    docker-rm
 GEN    docker-run
b9deb1f7ef337540a6d17731ef309768c22cc0fa9703ea68be819895e4d78732

Again, you will have a different hash shown for the container that is now running – you can check that it is running with docker ps -a:

docker ps -a

CONTAINER ID        IMAGE                COMMAND                  CREATED              STATUS              PORTS               NAMES
b9deb1f7ef33        demo_release:0.0.1   "/bin/sh -c 'exec ${B"   About a minute ago   Up About a minute                       demo_release

You can use the shortcut make docker-logs to display the logs from your container:

make docker-logs

 GEN    docker-logs
heart_beat_kill_pid = 1

So with the help of erlang.mk we have been able to bootstrap an erlang project, building and running a docker container in about 10 shell commands.

erlang in docker from scratch

When packaging an application as a Docker container it is too easy to just be lazy and put FROM debian (other distributions are available, replace debian with your distribution of choice). For sure it is going to work, but you have just included dozens of libraries and binaries that your application just does not need. An image that could be tens of megabytes is now at least several hundred – we are building containers not virtual machines here!

One of the things I like about Go is that typical application binaries are small with no runtime dependencies. Fewer dependencies mean less patching and security headaches. The less friction in the CI build cycle, the better. Go achieves this by having statically linked applications meaning that just one binary is necessary in ADD, and they are typically built from scratch (etcd as a good example).

Erlang was designed to be embedded in telecoms equipment, so we must be able to package applications in Docker with a small footprint too?

An example repository containing a regular erlang application that is packaged in a docker container from scratch. You will need both erlang installed and docker service running preferably on a Linux environment. The release needs to be built on Linux to be able to run on Linux because we are going include the ERTS.

On MacOS you might want to run shortishly/docker-erlang which will give you erlang and docker packaged together in a shell (brew install docker-machine if you don’t already have it):

docker run \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -t \
       -i \
       --rm \
       shortishly/docker-erlang \
       /bin/bash

Clone and build the erlang-in-docker-from-scratch repository, which contains a minimal erlang application that builds a release into the _rel directory:

git clone https://github.com/shortishly/erlang-in-docker-from-scratch.git eidfs
cd eidfs
make

At the end of make a standard erlang release for the eidfs application is now present in the _rel directory. To make it run inside a scratch container we need to include any runtime dependencies too. This is where mkimage comes in:

./bin/mkimage

The ./bin/mkimage script copies in any dynamic libraries that ERTS needs to run the erlang release:

'/lib/x86_64-linux-gnu/libc.so.6' -> '_rel/eidfs/lib/x86_64-linux-gnu/libc.so.6'
'/lib/x86_64-linux-gnu/libdl.so.2' -> '_rel/eidfs/lib/x86_64-linux-gnu/libdl.so.2'
'/lib/x86_64-linux-gnu/libm.so.6' -> '_rel/eidfs/lib/x86_64-linux-gnu/libm.so.6'
'/lib/x86_64-linux-gnu/libpthread.so.0' -> '_rel/eidfs/lib/x86_64-linux-gnu/libpthread.so.0'
'/lib/x86_64-linux-gnu/librt.so.1' -> '_rel/eidfs/lib/x86_64-linux-gnu/librt.so.1'
'/lib/x86_64-linux-gnu/libtinfo.so.5' -> '_rel/eidfs/lib/x86_64-linux-gnu/libtinfo.so.5'
'/lib/x86_64-linux-gnu/libutil.so.1' -> '_rel/eidfs/lib/x86_64-linux-gnu/libutil.so.1'
'/lib/x86_64-linux-gnu/libz.so.1' -> '_rel/eidfs/lib/x86_64-linux-gnu/libz.so.1'

It also copies /bin/sh so that we can run the release too. We can build a docker image for the release using the following command:

docker build \
       --build-arg REL_NAME=$(bin/release_name) \
       --build-arg ERTS_VSN=$(bin/system_version) \
       --pull=true \
       --no-cache=true \
       --force-rm=true \
       -t $(bin/release_name):$(bin/version) .

The $(bin/release_name), $(bin/system_version) and $(bin/version) are short escripts that respond with the release name, system ERTS version and the application version respectively.

Quite a lot of effort, what is the reward? Try docker images and look at the size of the resultant container:

REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
eidfs                      0.0.1               6748931f94e4        4 seconds ago       16.74 MB

We have a docker packaged erlang release in ~17MB. Lets run it!

docker run \
       --name $(bin/release_name) \
       -d \
       $(bin/release_name):$(bin/version)

Check the logs using docker logs $(bin/release_name) and you will see lots of application startup messages from SASL.

You might notice that the ENTRYPOINT used in the Dockerfile directly invokes erlexec. I have done this to reduce dependencies further so that the release, ERTS dynamic libraries, and /bin/bash only are present in the container.

FROM scratch
MAINTAINER Peter Morgan <peter.james.morgan@gmail.com>

ARG REL_NAME
ARG REL_VSN=1
ARG ERTS_VSN

ENV BINDIR /erts-${ERTS_VSN}/bin
ENV BOOT /releases/1/${REL_NAME}
ENV CONFIG /releases/${REL_VSN}/sys.config
ENV ARGS_FILE /releases/${REL_VSN}/vm.args

ENV TZ=GMT

ENTRYPOINT exec ${BINDIR}/erlexec \
           -boot_var /lib \
           -boot ${BOOT} \
           -noinput \
           -config ${CONFIG} \
           -args_file ${ARGS_FILE}

ADD _rel/${REL_NAME}/ /

A needle in a…

Just about everyone is using Docker. As a developer it radically simplifies how I package my code. However, in a tiered architecture, how is my presentation layer load balanced over my API layer? Out of the box Docker linking is to a single container and doesn’t provide load balancing. I’d like to introduce Haystack which provides automatic service discovery and load balancing of HTTP or WebSocket based services composed on Docker containers from a single node to a Swarm.

High Level Architecture
High Level Architecture

Haystack does this by monitoring the event stream of a single node or Swarm, noticing when containers start or stop (gracefully or otherwise). On startup a container is registered in Haystack’s DNS and any HTTP or WebSocket endpoints are automatically added to its load balancing proxy. Later when that same container stops, Haystack automatically removes it from the load balancer dropping its entry from DNS. Containers are organised into groups, partitioning services into tiers which may then be load balanced over all nodes with an Overlay network.

Haystack is itself a docker container which may be run on a single node or on every node in a swarm providing load balancing that is local to the node, or to other nodes in the cluster. We can create a simple Haystack environment using Docker Machine as follows:

# create a new docker machine environment called "default"
docker-machine create --driver virtualbox default

# use that environment in this shell
eval $(docker-machine env default)

# start Haystack connecting it to the Docker daemon
docker run -e DOCKER_HOST=${DOCKER_HOST} \
  -e DOCKER_KEY="$(cat ${DOCKER_CERT_PATH}/key.pem)" \
  -e DOCKER_CERT="$(cat ${DOCKER_CERT_PATH}/cert.pem)" \
  -e SHELLY_AUTHORIZED_KEYS="$(cat ~/.ssh/authorized_keys)" \
  -e DNS_NAMESERVERS=8.8.8.8:8.8.4.4 \
  --name=haystack \
  --publish=53:53/udp \
  --publish=80:80 \
  --publish=8080:8080 \
  --publish=22022:22 \
  --detach \
  shortishly/haystack

Haystack has 3 blocks of environmental configuration in the above command:

  • Lines 8 through 10 using DOCKER_HOST, DOCKER_KEY and DOCKER_CERT are enabling Haystack to communicate with the Docker Daemon using TLS.
  • The line containing SHELLY_AUTHORIZED_KEYS is optional. It copies your public keys into the Haystack container so that you can SSH directly into Haystack to perform maintenance or run tracing if things aren’t working OK. If you’re familiar with the Erlang Shell you can do so right now with ssh -p 22022 $(docker-machine ip default).
  • Finally the DNS_NAMESERVERS tells Haystack where to delegate any unresolved names. In the above example we’re using Google’s Public DNS. You can replace these entries with your own DNS if you wish.

Lets start a couple of nginx servers to be our web tier connecting them to the Haystack DNS service:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --detach nginx

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --detach nginx

The --dns options are telling the Nginx docker container to resolve its DNS using the Haystack container.

Haystack maintains a DNS SRV record for each endpoint exposed by a container. For example, starting an Nginx container will automatically populate an appropriate service record into Haystack’s DNS using the IP address and port exposed by that container. In addition, a DNS A record for that service is created which points to Haystack’s Load Balancer.

# nslookup -query=srv _http._tcp.nginx.services.haystack $(docker-machine ip default)

_http._tcp.nginx.services.haystack service = 100 100 80 c9pcjp7.containers.haystack.
_http._tcp.nginx.services.haystack	service = 100 100 80 c5162p7.containers.haystack.

# dig -query=a nginx.services.haystack $(docker-machine ip default)
nginx.services.haystack. 100	IN	A	172.17.0.2

Haystack has automatically created a DNS entry called nginx.services.haystack which is pointing to its own internal load balancing proxy. We can curl that address and the HTTP or WebSocket requests will be load balanced over the available Nginx instances:

docker run \
  --dns=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' haystack) \
  --tty --interactive --rm fedora /bin/bash

[root@a268fc00929d /]# curl http://nginx.services.haystack
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>

The DNS hierarchy created by Haystack is as follows:

Haystack DNS Hierarchy

Services that are available to be load balanced are published under the services.haystack namespace (for example, nginx.services.haystack). Haystack internally uses SRV records for each service instance, which Munchausen (the load balancing proxy) also references. Each docker container is also registered with a private Haystack name under the containers.haystack namespace.